Python piping a variable into logstash

Question

I am trying to invoke a logstash command from python2.7 with vars in the conf names, the full shell command I am trying to run is:

cat test.txt | sudo /usr/share/logstash/bin/logstash -f test.conf 

and here is my Python code:

listname = test
b = '/tmp/%s.txt' % (listname)
c = '/tmp/%s.conf' % (listname)
first = subprocess.Popen(['/bin/echo', b], stdout=subprocess.PIPE)
Second = subprocess.Popen(['/usr/share/logstash/bin/logstash -f', c], stdin=first.stdout)

I am having trouble inserting the listname variable as is created earlier in the code and I am also having trouble invoking logstash. Should I use the shell/subprocess to do this or is there a better way?

Answer 1

There are a number of issues with your code above. The main problems are likely that your listname should be a string (if test does not contain a string already) and that the syntax for your second Popen call is wrong (each argument should be a list entry of its own).

Also, if you need the second process to use sudo there are some extra issues, depending on whether you need to add a password or not.

Thus, depending on how to handle these issues, your code may look like one of the below alternatives:

from subprocess import Popen, PIPE

listname = 'test'
b = '/tmp/{}.txt'.format(listname)
c = '/tmp/{}.conf'.format(listname)

first = Popen(['/bin/cat', b], stdout=PIPE)

# Alt 1, No sudo
second = Popen(['/usr/share/logstash/bin/logstash', '-f', c], stdin=first.stdout)

# Alt 2, sudo with no password
second = Popen(['sudo', '/usr/share/logstash/bin/logstash', '-f', c], stdin=first.stdout)

# Alt 3, add sudo password (most complex)

# Alt 3.1, store password in file (bad)
sudo_password = "clear_text_password"

#Alt 3.2, ask user for password:
from getpass import getpass, getuser
sudo_password = getpass("[sudo] password for {}: ".format(getuser()))

# Since we need to add password at stdin, we cannot directly pipe output from "first"
# instead we set stdin to PIPE, and write sudo password, followed by output from "first"
# sudo -kSp '' ensures that password is read from stdin without prompt being displayed
second = Popen(['sudo', '-kSp', '', '/usr/share/logstash/bin/logstash', '-f', c], stdin=PIPE)

second.stdin.write(sudo_password + '
') # write password

# followed by output from "first".
for line in first.stdout:
    second.stdin.write(line)

second.stdin.close()

Hopefully this will help you find a suitable solution. If not, please specify your issues further.