A step in a path expression (that is not the last step) is returning atomic values, which is not permitted. For example, the following expression:

doc("catalog.xml")//name/substring(.,1,3)/replace(.,'A','a')

is not permitted because the second to last step, substring(.,1,.3), is returning atomic values. It can be rewritten as:

for $shortName in doc("catalog.xml")//name/substring(.,1,3)
return replace($shortName,'A','a')