Not tracking tensors returned by tf.reshape() for data sources other than MNIST

[khatchad]

Consider the following code:

# tf2_test_reshape.py
# From https://www.tensorflow.org/versions/r2.9/api_docs/python/tf/reshape

import tensorflow as tf


def f(a):
    pass


t1 = tf.ones([2, 3])
t2 = tf.reshape(t1, [6])
f(t2)

t2 should be a (reshaped) tensor, and the argument to f() should also be tracked as a tensor. Instead, I'm seeing this tensor analysis result:

[INFO] Tensor analysis: answer:
[Node: synthetic < PythonLoader, Ltensorflow/functions/reshape, do()LRoot; > Context: CallStringContext: [ script tf2_test_reshape.py.do()LRoot;@105 ], v2][{[D:Symbolic,n, D:Compound,[D:Constant,28, D:Constant,28]] of pixel}]
[Node: <Code body of function Lscript tf2_test_reshape.py> Context: CallStringContext: [ com.ibm.wala.FakeRootClass.fakeRootMethod()V@2 ], v245][{[D:Symbolic,n, D:Compound,[D:Constant,28, D:Constant,28]] of pixel}]
[Ret-V:Node: synthetic < PythonLoader, Ltensorflow/functions/ones, do()LRoot; > Context: CallStringContext: [ script tf2_test_reshape.py.do()LRoot;@100 ]][{[D:Symbolic,n, D:Compound,[D:Constant,28, D:Constant,28]] of pixel}]
[Node: synthetic < PythonLoader, Ltensorflow/functions/ones, do()LRoot; > Context: CallStringContext: [ script tf2_test_reshape.py.do()LRoot;@100 ], v5][{[D:Symbolic,n, D:Compound,[D:Constant,28, D:Constant,28]] of pixel}]

In the IR, v245 refers to the return value of tf.ones(). That's the only tensor in this file.

Regression

• There is a summary for tf.reshape(), but it calls copy_data() instead of read_data().
• There is special handling of tf.reshape() in the code, stemming from the MethodReference field com.ibm.wala.cast.python.ml.client.PythonTensorAnalysisEngine.reshape.

[khatchad]

Add this test works:

ponder-lab@2db36e2

I believe the problem is that the data sources are hard-coded:

ML/com.ibm.wala.cast.python.ml/source/com/ibm/wala/cast/python/ml/client/PythonTensorAnalysisEngine.java

Lines 606 to 610 in ddba21e

	TensorType mnistData = TensorType.mnistInput();
	Map<PointsToSetVariable, TensorType> init = HashMapFactory.make();
	for (PointsToSetVariable v : sources) {
	init.put(v, mnistData);
	}

[khatchad]

But, it could also have something to do with the way other APIs are being constructed. For example, the points-to set for tf.ones() is empty.

[khatchad]

Looking at the summary of tf.reshape(), I see that there's a data copy:

ML/com.ibm.wala.cast.python.ml/data/tensorflow.xml

Lines 379 to 388 in ddba21e

	<class name="reshape" allocatable="true">
	<method name="copy_data" descriptor="()LRoot;">
	<new def="x" class="Ltensorflow/examples/tutorials/mnist/dataset" />
	<return value="x" />
	</method>
	<method name="do" descriptor="()LRoot;" numArgs="3">
	<call class="LRoot" name="copy_data" descriptor="()LRoot;" type="virtual" arg0="arg0" def="x" />
	<return value="x" />
	</method>
	</class>

That may mean if the data source has something wrong with it, any copied data would also have the problem. Thus, the problem may not be with the tf.reshape() operation itself but rather with how data sources other than MNIST are constructed.

[khatchad]

That being said, copy_data() in the above summary doesn't use its argument.

[khatchad]

Thus, my best guess is that the problem involves a combination of the (new) XML summaries and the hard-coded initialization of the dataflow.