implement aspen.simplate.context

[chadwhitacre]

Want to explicitly import objects from aspen.context into simplates instead of placing them there magically.

[chadwhitacre]

I am torn.

from aspen.context import request may be more explicit and Pythonic and make kennethreitz happy. But PHP and Keystone show that people want teh magic in that last mile. And it won't make kennethreitz happy. He wants class Context. shudder

This is also really complicated to write due to the nine engines we support (can only use thread locals for some of them). I say to heck with aspen.context. Instead: better docs on the available objects.

[chadwhitacre]

I started using PyFlakes, which is rubbing my face in this.

And now I've thought of a simple way to implement this: pre-process the first simplate page to look for:

from aspen.context import request, response, website

The import target list would be stored, and only those objects requested would be placed in the execution context for pages 2 ff. This would work regardless of the backend, since we wouldn't be using threading locals or async library equivalent.

from aspen.context import ... would be commented out.

from aspen.context import * would obviously give you everything.

from aspen import context would give you the main context dictionary.

What about import aspen.context? I think that should clobber the actual aspen/context.py file and instead give you the main context dictionary, bound to aspen.context.

Waddya think, @kennethreitz? Also interested in @dcrosta's opinion.

[chadwhitacre]

Do the same for configure-aspen.py, I serpose.

[chadwhitacre]

Hrm, this is a little trickier than it sounds.

How does this behave?

from aspen.context import request
^L
from aspen.context import website
^L

There is no request object for page one, so importing it should be an ImportError.

Or this?

def foo(request):
    from aspen.context import response
    raise response
^L
from aspen.context import request
foo(request)
^L

Python import statements can appear anywhere and the names are defined in scope. The response object shouldn't be added to the global page one context, it should only be available inside foo.

We could do like future and require that it be the first line of the page (unless the first line is a future import? Do those work in exec?) I suppose we would then keep track of available names for each of the first two pages and raise ImportError if the requested name isn't among them. We could relax a little bit and allow "from aspen.context import ..." on lines other than the first as long as there's nothing before it except docstrings, comments and import statements. That's probably a good compromise.

[dcrosta]

I have some code which I believe correctly handles the most common types of imports using compiler.parse and a traversal of the AST. You could use this (or something like it) to detect the invalid import and raise an exception during compilation.

[chadwhitacre]

Heh. After writing that I went looking for that very code but got side-tracked w/ travis-ci.org integration (noticed you have keystone on there). Travis all set up, going to check out some AST madness.

!m @dcrosta

[dcrosta]

AST traversal >> source code scanning of any sort. You could probably do something more sophisticated than I am in Keystone with visitors (that would be able to more easily look inside of class or function definitions, for instance)

[chadwhitacre]

Do we know why compiler went away in 3.0? Is there a replacement?

[dcrosta]

Looks like you can use ast.parse and end up with something similar. Didn't even realize compiler was gone in Py3k.

[chadwhitacre]

Another idea: can I vary sys.modules for each execution context?

[dcrosta]

What type of thing is aspen.context? Is it the context.py module? (If so, what object is imported with from aspen.context import request?)

[chadwhitacre]

It's not strictly the context.py module, though the overload is intentional so that if anyone goes looking for aspen.context they can find doc in the context.py module.

It's supposed to be a "virtual module" that is magically available inside simplates. So instead of having magical names request, response, website, headers, qs, etc., you have a magical module that you import stuff from and it looks "normal," and if you dig into it you find docs really quickly.

[chadwhitacre]

Down to the wire on 1.0. One "punt" option here: implement a stupid re-based hack where we only match:

from aspen.context import ...

And probably only as the first line of each logic page.

That's low-hanging fruit, and syntax-wise it's a limiting case of the more robust AST-manipulation approach, so that we can implement the robust approach later on without breaking backwards compatibility.

We haven't even floated yet the idea of importing from aspen.context inside of a module (as opposed to a simplate). That would almost certainly require threadlocals or equivalent, which would cover us here as well and may in the end not be more complicated than AST manipulation.

[chadwhitacre]

If I implement this for @kennethreitz, will he appreciate it?

[dcrosta]

This may just be my opinion, but having from aspen.context import ... import a magical thing (i.e. not what you see when you look in context.py) is more surprising than simply having some thing (which you can find in a .py file, if you read the docs to find out what is being injected, as I do in Keystone) is more surprising. But I may just be resisting a change from how I do it here.

Why not make the "request" context simply fail to work when used at module-scope in the first page? It will have to be some sort of object which proxies to a thread-local (or something similar) in order to work in the first page anyway, but, as you suggest, there's a valid use case to define functions in the first page which are not used until the second page.

[chadwhitacre]

For the record, the reason this is important with PyFlakes is that it breaks the habituation of "oh crap it's red something is wrong." When so many red names are false positives, it re-trains you not to care about red. Before you know it Michael Fagan is sitting on the edge of your bed slugging Famous Grouse.

[pjz]

Can you not just allow the request to be imported but have it be None?

[chadwhitacre]

There's precedent in the future module for implementing some functionality outside of the usual interpreter mechanisms, while maintaining "a real module" that is simply for documentation and way-finding purposes.

[pjz]

see https://gist.github.com/pjz/5558820 for how to make a virtual module for exec'd code.

[chadwhitacre]

@pjz Okay, but that's still process-global. Request and response objects need to be local to each simplate request context. For thread-based network engines this would mean thread-locals. Not even sure what it would look like for an evented engine.

[chadwhitacre]

While writing the debug helper for algorithm.py, I thought of a way to implement this ticket:

1. Store a circular reference to the current context dict in a variable with a hacky name, like __aspen_context.
2. Modify the bytecode during simplate compilation so that from aspen.context import foo becomes foo = __aspen_context['foo'].

This has the advantage of reducing the variance from the behavior of the import statement to a minimum: both result in a STORE_FAST, as shown below, and the STORE_FAST will be at the same relative position in the bytecode either way. We can even perhaps implement the __aspen_context mapping in such a way that we raise ImportError instead of KeyError when accessed in this way. As @pjz suggests above, it would probably be fine to allow from aspen.context import request to simply be None in contexts other than simplate pages[1]. Then in aspen/context.py we can have request = None with a docstring explaining what it is and where it's available. This would even work as expected from a raw Python prompt.

$ python3 foo.py 
as import:
  7           0 LOAD_CONST               1 (0) 
              3 LOAD_CONST               0 (None) 
              6 IMPORT_NAME              0 (string) 
              9 STORE_FAST               0 (string) 
             12 LOAD_CONST               0 (None) 
             15 RETURN_VALUE         
as dict:
 11           0 LOAD_GLOBAL              0 (__aspen_context) 
              3 LOAD_CONST               1 ('string') 
              6 BINARY_SUBSCR        
              7 STORE_FAST               0 (string) 
             10 LOAD_CONST               0 (None) 
             13 RETURN_VALUE         
$ python2 foo.py 
as import:
  7           0 LOAD_CONST               1 (-1)
              3 LOAD_CONST               0 (None)
              6 IMPORT_NAME              0 (string)
              9 STORE_FAST               0 (string)
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE        
as dict:
 11           0 LOAD_GLOBAL              0 (__aspen_context)
              3 LOAD_CONST               1 ('string')
              6 BINARY_SUBSCR       
              7 STORE_FAST               0 (string)
             10 LOAD_CONST               0 (None)
             13 RETURN_VALUE        
$ cat foo.py 

from __future__ import print_function

import dis

def as_import():
    import string

__aspen_context = {'string': object}
def as_dict():
    string = __aspen_context['string']

print("as import:")
dis.dis(as_import)
print
print("as dict:")
dis.dis(as_dict)

[chadwhitacre]

+2 from @thomasboyt and @simon-weber in IRC.

[chadwhitacre]

I removed the pending network engine rework label from this ticket. The opcode implementation would be network engine-independent, and also we're intending to stop shipping a server.

[chadwhitacre]

We're getting rid of aspen/context.py in #379 (call). We can bring it back if we decide to use it here.

[chadwhitacre]

Revisited this on #496, still interested in it.

[chadwhitacre]

Though with #481 it attaches to simplates, not to aspen.

[chadwhitacre]

@pjz points out that importing request in the first page may not behave as expected in the second page.

[chadwhitacre]

Closing per AspenWeb/aspen.py#38 (comment).