Osh is not a shell. It is an executable implementing a language that supports the composition of commands through piping. The objects piped from one command to the next can be Python primitives (numbers, lists, maps, etc.), other useful types such as dates/times, or database rows; or represent various OS resources such as files, directories, and processes. Conversions between objects and strings simplify integration with the Unix environment. The commands included with osh manipulate objects, access databases, and execute commands remotely, including parallel execution on nodes in a cluster. In a single osh invocation, all commands run in a single process. Multithreading is used only when executing commands on a cluster.
Example: Suppose you have a cluster named fred, with nodes fred1, fred2, fred3. Each node has a database tracking work requests with a table named request. You can find the total number of open requests in each database as follows:
[jao@zack] osh @fred [ sql "select count(*) from request where state = 'open'" ] ^ out
('fred1', 1)
('fred2', 0)
('fred3', 5)
- osh: Invokes the osh executable.
- @fred [ ... ]: Specifies that the following command, delimited by [...] should be run on each node of the cluster named fred. (The osh configuration file, .oshrc, specifies how to connect to the nodes of the cluster.)
- sql "select count(*) from request where state = 'open'": Sql is an osh command that submits a query to a relational database. The query output is returned as a set of tuples.
- ^ out: ^ is the osh operator for piping objects from one command to the next In this case, the input objects are tuples resulting from execution of a SQL query on each node of the cluster. The out command renders each object as a string and prints it to stdout.
- Each output row identifies the node of origination (e.g. fred1, fred2), and contains a tuple from the database on that node. So ('fred3', 5) means that the database on node fred3 has 5 open requests.
Example, continued: Now suppose you want to find the total number of open requests across the cluster. You can pipe the tuples into an aggregation command:
[jao@zack] osh @fred [ sql "select count(*) from request where state = 'open'" ] ^ agg 0 'total, node, count: total + count' $ 6
- agg: agg is the aggregation command. Tuples from across the cluster are piped into the agg command, which will accumulate results from all inputs.
- 0: This use of agg will maintain a total, which is initialized to 0.
- 'total, node, count: total + count': This specifies an aggregation function. total is the running total, which was initialized to 0. node and count come from the sql command executed on each node of the cluster. total + count accumulates the counts from each node.
- $: An alternative to ^ out that can be used at the end of a command only.
- 6: The total of the counts from across the cluster.
More information:
License: GPL
Release history
Tutorial (TBD)
User guide
Download
Software with similar goals
jao@geophile.com
from: http://geophile.com/osh/
String manipulation is very useful and very widely used in every language. Often, programmers are required to break down strings and examine them closely. For example, in my articles on IRC (http://www.devshed.com/c/a/Python/Python-and-IRC/ and http://www.devshed.com/c/a/Python/Basic-IRC-Tasks/), I used the split method to break down commands to make working with them easier. In my article on sockets (http://www.devshed.com/c/a/Python/Sockets-in-Python/), I used regular expressions to look through a website and extract a currency exchange rate.
This article will take a look at the various methods of manipulating strings, covering things from basic methods to regular expressions in Python. String manipulation is a skill that every Python programmer should be familiar with.
String Methods
The most basic way to manipulate strings is through the methods that are build into them. We can perform a limited number of tasks to strings through these methods. Open up the Python interactive interpreter. Let's create a string and play around with it a bit.
>>> test = 'This is just a simple string.'
Let's take a fast detour and use the len function. It can be used to find the length of a string. I'm not sure why it's a function rather than a method, but that's a whole nother issue:
>>> len ( test )
29
All right, now let's get back to those methods I was talking about. Let's take our string and replace a word using the replace method:
>>> test = test.replace ( 'simple', 'short' )
>>> testa
'This is just a short string.'
Now let's count the number of times a given word, or, in this case, character, appears in a string:
>>> test.count ( 'r' )
2
We can find characters or words, too:
>>> test.find ( 'r' )
18
>>> test [ 18 ]
'r'Splitting a string is something I find myself doing often. The split method is used for this:
>>> test.split()
['This', 'is', 'just', 'a', 'short', 'string.']We can choose the point that we split it at:
>>> test.split ( 'a' )
['This is just ', ' short string.']Rejoining our split string can be done using the join method:
>>> ' some '.join ( test.split ( 'a' ) )
'This is just some short string.'We can play around with the case of letters in our string, too. Let's make it all upper case:
>>> test.upper()
'THIS IS JUST A SHORT STRING.'Now let's make it lowercase:
>>> test.lower()
'this is just a short string.'Let's capitalize only the first letter of the lowercase string:
>>> test.lower().capitalize()
'This is just a short string.'We can also use the title method. This capitalizes the first letter in each word:
>>> test.title()
'This Is Just A Short String.'Trading case is possible:
>>> test.swapcase()
'tHIS IS JUST A SHORT STRING.'We can run a number of tests on strings using a few methods. Let's check to see whether a given string is all upper case:
>>> 'UPPER'.isupper()
True
>>> 'UpPEr'.isupper()
FalseLikewise, we can check to see whether a string contains only lower case characters:
>>> 'lower'.islower()
True
>>> 'Lower'.islower()
FalseChecking whether a string looks like a title is simple, too:
>>> 'This Is A Title'.istitle()
True
>>> 'This is A title'.istitle()
FalseWe can check whether a string is alphanumeric:
>>> 'aa44'.isalnum()
True
>>> 'a$44'.isalnum()
FalseIt is also possible to check whether a string contains only letters:
>>> 'letters'.isalpha()
True
>>> 'letters4'.isalpha()
FalseHere's how you check whether a string contains only numbers:
>>> '306090'.isdigit()
True
>>> '30-60-90 Triangle'.isdigit()
FalseWe can also check whether a string only contains spaces:
>>> ' '.isspace()
True
>>> ''.isspace()
FalseSpeaking of spaces, we can add spaces on either side of a string. Let's add spaces to the right of a string:
>>> 'A string.'.ljust ( 15 )
'A string. 'To add spaces to the left of a string, the rjust method is used:
>>> 'A string.'.rjust ( 15 )
' A string.'The center method is used to center a string in spaces:
>>> 'A string.'.center ( 15 )
' A string. 'We can strip spaces on either side of a string:
>>> 'String.'.rjust ( 15 ).strip()
'String.'
>>> 'String.'.ljust ( 15 ).rstrip()
'String.'Regular expressions are a very powerful tool in any language. They allow patterns to be matched against strings. Actions such as replacement can be performed on the string if the regular expression pattern matches. Python's module for regular expressions is the re module. Open the Python interactive interpreter, and let's take a closer look at regular expressions and the re module:
>>> import re
Let's create a simple string we can use to play around with:
>>> test = 'This is for testing regular expressions in Python.'
I spoke of matching special patterns with regular expressions, but let's start with matching a simple string just to get used to regular expressions. There are two methods for matching patterns in strings in the re module: search and match. Let's take a look at search first. It works like this:
>>> result = re.search ( 'This', test )
We can extract the results using the group method:
>>> result.group ( 0 )
'This'You're probably wondering about the group method right now and why we pass zero to it. It's simple, and I'll explain. You see, patterns can be organized into groups, like this:
>>> result = re.search ( '(Th)(is)', test )
There are two groups surrounded by parenthesis. We can extract them using the group method:
>>> result.group ( 1 )
'Th'
>>> result.group ( 2 )
'is'Passing zero to the method returns both of the groups:
>>> result.group ( 0 )
'This'The benefit of groups will become more clear once we work our way into actual patterns. First, though, let's take a look at the match function. It works similarly, but there is a crucial difference:
>>> result = re.match ( 'This', test )
>>> print result
<_sre.SRE_Match object at 0x00994250>
>>> print result.group ( 0 )
'This'
>>> result = re.match ( 'regular', test )
>>> print result
NoneNotice that None was returned, even though 鈥渞egular鈥?is in the string. If you haven't figured it out, the match method matches patterns at the beginning of the string, and the search function examines the whole string. You might be wondering if it's possible, then, to make the match method match 鈥渞egular,鈥?since it's not at the beginning of the string. The answer is yes. It's possible to match it, and that brings us into patterns.
The character 鈥?鈥?will match any character. We can get the match method to match 鈥渞egular鈥?by putting a period for every letter before it. Let's split this up into two groups as well. One will contain the periods, and one will contain 鈥渞egular鈥?
>>> result = re.match ( '(....................)(regular)', test )
>>> result.group ( 0 )
'This is for testing regular'
>>> result.group ( 1 )
'This is for testing '
>>> result.group ( 2 )
'regular'Aha! We matched it! However, it's ridiculous to have to type in all those periods. The good news is that we don't have to do that. Take a look at this and remember that there are twenty characters before 鈥渞egular鈥?
>>> result = re.match ( '(.{20})(regular)', test )
>>> result.group ( 0 )
'This is for testing regular'
>>> result.group ( 1 )
'This is for testing '
>>> result.group ( 2 )
'regular'That's a lot easier. Now let's look at a few more patterns. Here's how you can use brackets in a more advanced way:
>>> result = re.match ( '(.{10,20})(regular)', test )
>>> result.group ( 0 )
'This is for testing regular'
>>> result = re.match ( '(.{10,20})(testing)', test )
'This is for testing'By entering two arguments, so to speak, you can match any number of characters in a range. In this case, that range is 10-20. Sometimes, however, this can cause undesired behavior. Take a look at this string:
>>> anotherTest = 'a cat, a dog, a goat, a person'
Let's match a range of characters:
>>> result = re.match ( '(.{5,20})(,)', anotherTest )
>>> result.group ( 1 )
'a cat, a dog, a goat'What if we only want 鈥渁 cat鈥?though? This can be done with appending 鈥?鈥?to the end of the brackets:
>>> result = re.match ( '(.{5,20}?)(,)', anotherTest )
>>> result.group ( 1 )
'a cat'Appending a question mark to something makes it match as few characters as possible. A question mark that does that, though, is not to be confused with this pattern:
>>> anotherTest = '012345'
>>> result = re.match ( '01?', anotherTest )
>>> result.group ( 0 )
'01'
>>> result = re.match ( '0123456?', anotherTest )
>>> result.group ( 0 )
'012345'As you can see with the example, the character before a question mark is optional. Next is the 鈥?鈥?pattern. It matches one or more of the characters it follows, like this:
>>> anotherTest = 'Just a silly string.'
>>> result = re.match ( '(.*)(a)(.*)(string)', anotherTest )
>>> result.group ( 0 )
'Just a silly string'However, take a look at this:
>>> anotherTest = 'Just a silly string. A very silly string.'
>>> result = re.match ( '(.*)(a)(.*)(string)', anotherTest )
>>> result.group ( 0 )
'Just a silly string. A very silly string'What if, however, we want to only match the first sentence? If you've been following along closely, you'll know that 鈥?鈥?will, again, do the trick:
>>> result = re.match ( '(.*?)(a)(.*?)(string)', anotherTest )
>>> result.group ( 0 )
'Just a silly string'As I mentioned earlier, though, 鈥?鈥?doesn't have to match anything:
>>> result = re.match ( '(.*?)(01)', anotherTest )
>>> result.group ( 0 )
'01'What if we want to skip past the first two characters? This is possible by using 鈥?鈥? which is similar to 鈥?鈥? except that it matches at least one character:
>>> result = re.match ( '(.+?)(01)', anotherTest )
>>> result.group ( 0 )
'0101'We can also match a range of characters. For example, we can match only the first four letters of the alphabet:
>>> anotherTest = 'a101'
>>> result = re.match ( '[a-d]', anotherTest )
>>> print result
<_sre.SRE_Match object at 0x00B47B10>
>>> anotherTest = 'q101'
>>> result = re.match ( '[a-d]', anotherTest )
>>> print result
NoneWe can also match one of a few patterns using 鈥渱鈥?:
>>> testA = 'a'
>>> testB = 'b'
>>> result = re.match ( '(a|b)', testA )
>>> print result
<_sre.SRE_Match object at 0x00B46D60>
>>> result = re.match ( '(a|b)', testB )
>>> print result
<_sre.SRE_Match object at 0x00B46E60>Finally, there are a number of special sequences. 鈥淺A鈥?matches at the start of a string. 鈥淺Z鈥?matches at the end of a string. 鈥淺d鈥?matches a digit. 鈥淺D鈥?matches anything but a digit. 鈥淺s鈥?matches whitespace. 鈥淺S鈥?matches anything but whitespace.
We can name our groups:
>>> nameTest = 'hot sauce'
>>> result = re.match ( '(?P<one>hot)', nameTest )
>>> result.group ( 'one' )
'hot'We can compile patterns to use them multiple times with the re module, too:
>>> ourPattern = re.compile ( '(.*?)(the)' )
>>> testString = 'This is the dog and the cat.'
>>> result = ourPattern.match ( testString )
>>> result.group ( 0 )
'This is the'Of course, you can do more than match and extract substrings. You can replace things, too:
>>> someString = 'I have a dream.'
>>> re.sub ( 'dream', 'dog', someString )
'I have a dog.'On a final note, you should not use regular expressions to match or replace simple strings.
Conclusion
Now you have a basic knowledge of string manipulation in Python behind you. As I explained at the very beginning of the article, string manipulation is necessary to many applications, both large and small. It is used frequently, and a basic knowledge of it is critical.
from:http://www.devshed.com/c/a/Python/String-Manipulation/