This document provides answers to some of the frequently asked questions about TensorFlow. If you have a question that is not covered here, you might find an answer on one of the TensorFlow community resources.
[TOC]
Yes! TensorFlow gained support for distributed computation in version 0.8. TensorFlow now supports multiple devices (CPUs and GPUs) in one or more computers.
As of the 0.6.0 release timeframe (Early December 2015), we do support Python 3.3+.
See also the API documentation on building graphs.
In the TensorFlow Python API, a, b, and c are
Tensor objects. A Tensor object is
a symbolic handle to the result of an operation, but does not actually hold the
values of the operation's output. Instead, TensorFlow encourages users to build
up complicated expressions (such as entire neural networks and its gradients) as
a dataflow graph. You then offload the computation of the entire dataflow graph
(or a subgraph of it) to a TensorFlow
Session, which is able to execute the
whole computation much more efficiently than executing the operations
one-by-one.
The supported device names are "/device:CPU:0" (or "/cpu:0") for the CPU
device, and "/device:GPU:i" (or "/gpu:i") for the ith GPU device.
To place a group of operations on a device, create them within a
with tf.device(name): context. See
the how-to documentation on
using GPUs with TensorFlow for details of how
TensorFlow assigns operations to devices, and the
CIFAR-10 tutorial for an example model that
uses multiple GPUs.
TensorFlow supports a variety of different data types and tensor shapes. See the ranks, shapes, and types reference for more details.
See also the API documentation on running graphs.
Feeding is a mechanism in the TensorFlow Session API that allows you to
substitute different values for one or more tensors at run time. The feed_dict
argument to Session.run() is a
dictionary that maps Tensor objects to
numpy arrays (and some other types), which will be used as the values of those
tensors in the execution of a step.
Often, you have certain tensors, such as inputs, that will always be fed. The
tf.placeholder() op allows you
to define tensors that must be fed, and optionally allows you to constrain
their shape as well. See the
beginners' MNIST tutorial for an
example of how placeholders and feeding can be used to provide the training data
for a neural network.
If t is a Tensor object,
t.eval() is shorthand for
sess.run(t) (where sess is the
current default session. The
two following snippets of code are equivalent:
# Using `Session.run()`.
sess = tf.Session()
c = tf.constant(5.0)
print sess.run(c)
# Using `Tensor.eval()`.
c = tf.constant(5.0)
with tf.Session():
print c.eval()In the second example, the session acts as a
context manager,
which has the effect of installing it as the default session for the lifetime of
the with block. The context manager approach can lead to more concise code for
simple use cases (like unit tests); if your code deals with multiple graphs and
sessions, it may be more straightforward to make explicit calls to
Session.run().
Sessions can own resources, such as
variables,
queues, and
readers; and these resources can use
a significant amount of memory. These resources (and the associated memory) are
released when the session is closed, by calling
Session.close().
The intermediate tensors that are created as part of a call to
Session.run() will be freed at or before the
end of the call.
The TensorFlow runtime parallelizes graph execution across many different dimensions:
- The individual ops have parallel implementations, using multiple cores in a CPU, or multiple threads in a GPU.
- Independent nodes in a TensorFlow graph can run in parallel on multiple devices, which makes it possible to speed up CIFAR-10 training using multiple GPUs.
- The Session API allows multiple concurrent steps (i.e. calls to Session.run() in parallel. This enables the runtime to get higher throughput, if a single step does not use all of the resources in your computer.
TensorFlow is designed to support multiple client languages. Currently, the best-supported client language is Python. The C++ client API provides an interface for launching graphs and running steps; we also have an experimental API for building graphs in C++.
We would like to support more client languages, as determined by community interest. TensorFlow has a C-based client API that makes it easy to build a client in many different languages. We invite contributions of new language bindings.
TensorFlow supports multiple GPUs and CPUs. See the how-to documentation on using GPUs with TensorFlow for details of how TensorFlow assigns operations to devices, and the CIFAR-10 tutorial for an example model that uses multiple GPUs.
Note that TensorFlow only uses GPU devices with a compute capability greater than 3.5.
The reader and
queue classes provide special operations that
can block until input (or free space in a bounded queue) becomes
available. These operations allow you to build sophisticated
input pipelines, at the cost of making the
TensorFlow computation somewhat more complicated. See the how-to documentation
for
using QueueRunner objects to drive queues and readers
for more information on how to use them.
See also the how-to documentation on variables and variable scopes, and the API documentation for variables.
A variable is created when you first run the
tf.Variable.initializer
operation for that variable in a session. It is destroyed when that
session is closed.
Variables allow concurrent read and write operations. The value read from a
variable may change if it is concurrently updated. By default, concurrent
assigment operations to a variable are allowed to run with no mutual exclusion.
To acquire a lock when assigning to a variable, pass use_locking=True to
Variable.assign().
See also the
TensorShape API documentation.
In TensorFlow, a tensor has both a static (inferred) shape and a dynamic (true)
shape. The static shape can be read using the
tf.Tensor.get_shape()
method: this shape is inferred from the operations that were used to create the
tensor, and may be
partially complete. If the static
shape is not fully defined, the dynamic shape of a Tensor t can be
determined by evaluating tf.shape(t).
The tf.Tensor.set_shape() method updates
the static shape of a Tensor object, and it is typically used to provide
additional shape information when this cannot be inferred directly. It does not
change the dynamic shape of the tensor.
The tf.reshape() operation creates
a new tensor with a different dynamic shape.
It is often useful to build a graph that works with variable batch sizes, for example so that the same code can be used for (mini-)batch training, and single-instance inference. The resulting graph can be saved as a protocol buffer and imported into another program.
When building a variable-size graph, the most important thing to remember is not
to encode the batch size as a Python constant, but instead to use a symbolic
Tensor to represent it. The following tips may be useful:
-
Use
batch_size = tf.shape(input)[0]to extract the batch dimension from aTensorcalledinput, and store it in aTensorcalledbatch_size. -
Use
tf.reduce_mean()instead oftf.reduce_sum(...) / batch_size. -
If you use placeholders for feeding input, you can specify a variable batch dimension by creating the placeholder with
tf.placeholder(..., shape=[None, ...]). TheNoneelement of the shape corresponds to a variable-sized dimension.
See the graph visualization tutorial.
Add summary ops to your TensorFlow graph, and use a
SummaryWriter to write
these summaries to a log directory. Then, start TensorBoard using
python tensorflow/tensorboard/tensorboard.py --logdir=path/to/log-directory
For more details, see the Summaries and TensorBoard tutorial.
You can change TensorBoard to serve on localhost rather than '0.0.0.0' by the flag --host=localhost. This should quiet any security warnings.
See also the how-to documentation for adding a new operation to TensorFlow.
There are two main options for dealing with data in a custom format.
The easier option is to write parsing code in Python that transforms the data
into a numpy array, then feed a
tf.placeholder() a tensor with
that data. See the documentation on
using placeholders for input for
more details. This approach is easy to get up and running, but the parsing can
be a performance bottleneck.
The more efficient option is to add a new op written in C++ that parses your data format. The guide to handling new data formats has more information about the steps for doing this.
The TensorFlow op registration mechanism allows you to define inputs that are a single tensor, a list of tensors with the same type (for example when adding together a variable-length list of tensors), or a list of tensors with different types (for example when enqueuing a tuple of tensors to a queue). See the how-to documentation for adding an op with a list of inputs or outputs for more details of how to define these different input types.
The TensorFlow Python API adheres to the
PEP8 conventions.* In
particular, we use CamelCase names for classes, and snake_case names for
functions, methods, and properties. We also adhere to the
Google Python style guide.
The TensorFlow C++ code base adheres to the Google C++ style guide.
(* With one exception: we use 2-space indentation instead of 4-space indentation.)