SRL Archive - Help and Support

snip

From: Sin
I have to write a function that returns the average amount of words per sentence, given a text. The text must be split using the definition of a sentence (anything ending in '?', '!', '.'). So far I've had no problems. However, once I split the text, there's always a left over '\n' that causes an extra length to be added to the sentence array which skews the whole result:
…

lets do a lil' code golfing! :p

def avg_sentence_length(text):
  text = ''.join(text)
  sentences = text.replace('?','.').replace('!','.').split('.')
  sums = [len(x.split()) for x in sentences if x.strip() != '']
  return sum(sums) / float(len(sums))

if __name__ == '__main__':
  text = ['The time has come, the Walrus said\n',
          'To talk of many things: of shoes - and ships - and sealing wax,\n',
          'Of cabbages; and kings.\n',
          'And why the sea is boiling hot;\n',
          'and whether pigs have wings.\n']
  print avg_sentence_length(text)

>> 18.5

From: slacky

def avg_sentence_length(text):
  text = ''.join(text)
  sentences = text.replace('?','.').replace('!','.').split('.')
  sums = [len(x.split()) for x in sentences if x.strip() != '']
  return sum(sums) / float(len(sums ))

if __name__ == '__main__':
  text = ['The time has come, the Walrus said\n',
          'To talk of many things: of shoes - and ships - and sealing wax,\n',
          'Of cabbages; and kings.\n',
          'And why the sea is boiling hot;\n',
          'and whether pigs have wings.\n']
  print avg_sentence_length(text)

>> 18.5

The function I posted gets 18.5 as well, however in the function description given to us, it's 17.5

From: Sin
The function I posted gets 18.5 as well, however in the function description given to us, it's 17.5

No it doesnt (not the code you posted here at least). Your code yields 12 (python 2.7), and 12.3333… in python 3.x

18.5 is the average.
Just count it your self:
- first part = 25 "words"
- sec part = 12 "words"
(12+25) / 2 = 18.5

From: slacky
No it doesnt (not the code you posted here at least). Your code yields 12 (python 2.7), and 12.3333… in python 3.x

Ah yeah, realized that just now. I figured out why that happens though, when splitting it seems to like to identify '-' as a word, which is why it skews the sentence word length.

@slacky

Edit:
I've figured out avg_sentence_length by just replacing all '-' with a ''.

def avg_sentence_complexity(text):
    """ (list of str) -> float

    Precondition: text contains at least one sentence.    

    A sentence is defined as a non-empty string of non-terminating
    punctuation surrounded by terminating punctuation or beginning or
    end of file. Terminating punctuation is defined as !?.
    Phrases are substrings of sentences, separated by one or more of ,;:

    Return the average number of phrases per sentence in text.

    >>> text = ['The time has come, the Walrus said\n',
         'To talk of many things: of shoes - and ships - and sealing wax,\n',
         'Of cabbages; and kings.\n',
         'And why the sea is boiling hot;\n',
         'and whether pigs have wings.\n']
    >>> avg_sentence_complexity(text)
    3.5
    """
    c = ''.join(text)
    sentences = c.replace('?','.').replace('-','').replace('!','.').split('.')
    new_str = []
    for x in sentences:
        new_str.append(x.replace(';',':').replace('-','').replace(',',':').split(':'))
    
    print(new_str)

However every single time I end up with a a third element in new_str with just '\n' in it. It has no string length, however it messes up the whole result when I try to divide the total number of phrases by each sentence.
(number of phrases overall / number of sentences)

From: Sin
Ah yeah, realized that just now. I figured out why that happens though, when splitting it seems to like to identify '-' as a word, which is why it skews the sentence word length.

This should do what you want, it will only count words:

import re

def avg_sentence_length(text):
  text = ''.join(text)
  sentences = text.replace('?','.').replace('!','.').split('.')
  sums = [x for x in [len(re.findall(r"\w+", x)) for x in sentences] if x!=0]
  return sum(sums) / float(len(sums))



text = ['The time has come, the Walrus said\n',
        'To talk of many things: of shoes - and ships - and sealing wax,\n',
        'Of cabbages; and kings.\n',
        'And why the sea is boiling hot;\n',
        'and whether pigs have wings.\n']
print(avg_sentence_length(text))

>> 17.5

Edit: For your edit in the post above: Just look at how I did it in my code to avoid that "\n" at the end, or just use the solution I just provided.

From: slacky
This should do what you want, it will only count words:

import re

def avg_sentence_length(text):
  text = ''.join(text)
  sentences = text.replace('?','.').replace('!','.').split('.')
  sums = [x for x in [len(re.findall(r"\w+", x)) for x in sentences] if x!=0]
  return sum(sums) / float(len(sums))



text = ['The time has come, the Walrus said\n',
        'To talk of many things: of shoes - and ships - and sealing wax,\n',
        'Of cabbages; and kings.\n',
        'And why the sea is boiling hot;\n',
        'and whether pigs have wings.\n']
print(avg_sentence_length(text))

>> 17.5

Edit: For your edit in the post above: Just look at how I did it in my code to avoid that "\n" at the end, or just use the solution I just provided.

Im not too sure we're allowed to use re ;/

From: Sin
Im not too sure we're allowed to use re ;/

gad…

junkchars = """!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~ \t\n\r\v\f\0"""   

def avg_sentence_length(text):
  text = ''.join(text)
  sentences = text.replace('?','.').replace('!','.').split('.')
  countwords = lambda x: sum(1 for v in x.split() if v.strip(junkchars) != '')
  sums = [x for x in [countwords(x) for x in sentences] if x!=0]
  return sum(sums) / float(len(sums))


text = ['The time has come, the Walrus said\n',
        'To talk of many things: of shoes - and ships - and sealing wax,\n',
        'Of cabbages; and kings.\n',
        'And why the sea is boiling hot;\n',
        'and whether pigs have wings.\n']

print avg_sentence_length(text)

>>> 17.5
Can be done differently, where we bases us (somewhat) on how \w+ works, by checking for alphanum, -, and _.. anything else can be considered space.. but W/E

- Good night.

From: slacky
gad…

junkchars = """!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~ \t\n\r\v\f\0"""   

def avg_sentence_length(text):
  text = ''.join(text)
  sentences = text.replace('?','.').replace('!','.').split('.')
  countwords = lambda x: sum(1 for v in x.split() if v.strip(junkchars) != '')
  sums = [x for x in [countwords(x) for x in sentences] if x!=0]
  return sum(sums) / float(len(sums))


text = ['The time has come, the Walrus said\n',
        'To talk of many things: of shoes - and ships - and sealing wax,\n',
        'Of cabbages; and kings.\n',
        'And why the sea is boiling hot;\n',
        'and whether pigs have wings.\n']

print avg_sentence_length(text)

>>> 17.5
Can be done differently, where we bases us (somewhat) on how \w+ works, by checking for alphanum, -, and _.. anything else can be considered space.. but W/E

- Good night.

Slacky, would you mind telling me what these two lines do?

countwords = lambda x: sum(1 for v in x.split() if v.strip(junkchars) != '')
  sums = [x for x in [countwords(x) for x in sentences] if x!=0]

From: Sin
Slacky, would you mind telling me what these two lines do?

countwords = lambda x: sum(1 for v in x.split() if v.strip(junkchars) != '')
sums = [x for x in [countwords(x) for x in sentences] if x!=0]

Instead of me explaning it, i'll let this post do it for me.

However I can take the time to roughly explain it codewise (untested):

#countwords = lambda x: sum(1 for v in x.split() if v.strip(junkchars) != '')
def countwords(x):
  result = 0;
  for v in x.split():
    if v.strip(junkchars) != '': #strip = remove all right (and left) characters that fit any of the given chars (order insensitive) !!not the same as simbas trim!!
      result += 1

lambda = anonymous function.
Sum(iterable) is a function that takes an iterable, so I just pass it a generator that spits out 1 in each iter if "v.strip(junkchars) != ''".

#sums = [count for count in [countwords(x) for x in sentences] if count!=0]
sums = []
for x in sentences:
  count = countwords(x)
  if count != 0:
    sums.append(count)

List comprehension (EG [x for x in iterable]) is highly optimized in CPython, and therefor the preferred way for simple loops like these.