Notes on Tensorflow Programmers Guide.

Tensorflow - notes on programmers guide.

1. Basic concepts:

TensorFlow separates definition of compuations from their execution. It first assemble a graph (tf.graph) and then use a session to execute operations in the graph (tf.session). A graph can be separated into subgraphs and execution can be computed in distributed manner.
  • Tensor: n-dimensional array (not mathematically rigorous).
  • Nodes (graphs): operators, variables and constants.
  • Edges (graphs): tensors.
  • tf.Session(): it encapsulates the environment in which operation objects are executed, and tensor  objects are evaluated. It also allocate memory to store the current values of varaibles. 
  • tf.Graph(): creates a graph. tf.get_default_graph() acquires the default graph. Mixing this user created graph with default graph is prone to errors.
  • tf.constant(): sets a specific value and cannot be changed. Note that this makes loading graphs expensive when constants are big.
  • tf.Variable() or tf.get_variable(): a class with many ops such as init op, read op, write op and more. tf.get_variable() is usually better. Initialization must be done!
  • tf.placeholder(): a class where we can later supply their own data when they need to execute the computation. Data is supplemented to placeholders using dictionary.
Tensorboard is a visualization tool (not only the graphs) that makes it easier to understand, debug, and optimize TensorFlow programs. Run in your terminal to use it.
$ python [yourprogram].py
$ tensorboard --logdir="./graphs" --port XXXX

Questions and tips (to be updated):
  1. What if I want to build more than one graph? This is a bad idea because multiple graphs require multiple sessions, each will try to use all available resources by default. Furthermore, we cannot pass data between them without passing them through python, which does not work in distributed sense. Just better to have disconnected subgraphs within one graph!
  2. Never mix default graph and user created graph. It is just error prone.
  3. Use tf data types instead of numpy and python as much as possible. TensorFlow has to infer pyhton native types and numpy arrays are not GPU compatible. 
  4. Use tf.constant() only for preimitive types and do not make it big.
  5. Lazing loading must always be avoided.
  6. "Datasets" are preferable method of streaming data into a model. Use placeholders for only simple experiments.

2. High level API - Eager Execution.

Eager execution is an impreative programming enrionement that evaluates operations immediately instead of building graph and executing them. Mainly it is useful to debug.
  • tf.enable_eager_execution(): it starts eager execution and put them to the beginning.
  • import tensorflow.contrib.eager as tfe: tfe module contatins symbols available to both eager and graph execution environements and is useful for coding to work with graphs.
Personally I should not use it because (1) overheads on smaller operation, (2) no distributed support, (3) not all TF APIs are eager compatible, (4) break components of the graph and debug instead of this and (5) you don't learn tensorflow which I believe will remian declarative programming language.

3. High level API - Importing data: tf.data API.

The tf.data API has two abstractions (1) tf.data.Dataset (sequence of elements where each element is an training example, and it can create a dataset from tf.Tensor objects and also from tf.data.Dataset objects) and (2) tf.data.Iterator (a main way to extract elements from a dataset).

Dataset structure: a dataset comprieses elements with the same structure. An element comtains tensor objects called components.
  • tf.data.Dataset.from_tensor_slices(): creates a dataset object with a type and shape.
  • tf.data.Dasaset.zip(): combines dataset objects.
Iterator: a class that helps access elements from that dataset. We have one-shot, initializable, reinitializable and feedable iterators.
  • One-shot: simplest & only support iterating once through a dataset. Only type that is easily usable with Estimator APIs.
  • Initializable: enables parameterization of the definition in the dataset using placeholder.
  • Reinitializable: enables initiliazable iterator across multiple dataset objects including training and validation sets within the same iterator object.
  • Feedable: the same as reinitializable but does not require to initialize the iterator from the start to switch betwwen iterators.
How to then read input data?
  • Numpy arrays: load numpy array and save them as dataset object. Then use one of the iterators.
  • TFRecord data: the tf.data API supports a variety of file formats so that you can process large datasets that do not fit in memory. The tf.data.TFRecordDataset class enables streaming over the contents of one or more TFRecordfiles as part of an input pipeline.
  • Text data: a high level API (tf.data.TextLineDataset) that enables easy handling of text files.
Out-of-range-error when using iterator?
  • always use try-except block while iterating for training.
How to save iterator state?
  • tf.contrib.data.make_saveable_from_iterator function to create an object from an iterator. Then, tf.train.Saver variable can then be used. 
Input processing: Dataset.map() can be used with a parsing function in order to process the input images.

How to decode image data and resize it?
  • tf.image.decode_image and tf.image.resize_images can be used as parse function to tf.dataset.map. It is to convert images of different sizes to a common size, so that they may be batched into a fixed size.
How to parse tf.Example protocol buffer messages?
  • tf.train.Example protocol buffer messages from aTFRecord-format file can be extracted (e.g. image and labels). tf.dataset.map used with a function that does it. A scalar string can be transformed into a pair of a scalar string and a scalar integer, representing an image and its label as an example.
How to arbitrarily apply python logic with tf.py_func()?
  • tf.py_func() operation in a Dataset.map() transformation can be done if we want to call upon external Python libraries to parse our input data.
How to batch dataset elements?
  • Dataset.batch() can batch stacks of n consecutive elements of a dataset into a single element. 
  • Dataset.padded_batch() can be used when tensors have varying size.
How to process multiple epochs?
  • Dataset.repeat() create a dataset that repeats its input for 10 epochs.
  • tf.errors.OutOfRangeError with try-except wrap during train must be done when using this.
How to randomly shuffle input data?
  • Dataset.shuffle() randomly shuffles the input datset with a fixed-size buffer. Next elements are randomly (with uniform) taken from that buffer.

4. High level API - Estimators.

Estimators encapsulate the following actions: training, evaluation, prediction and export for serving. There exists pre-made or custom made estimators based on calss tf.estimator.Estimator class.

Using pre-made estimators involves following steps.
  1. Data set importing functions.
  2. Defining the feautre columns.
  3. Instantiate the relevant pre-made Estimator.
  4. Call a training, evaluation or inference method.
Creating a custom made estimator involves following steps.
  1. Data set importing functions
  2. Defining the feature columns.
  3. A model function.
  4. Training, evaluation and prediction functions.
  5. Instantiating the Estimator and calling a training, evaluation or inference method.
Whether to use low level API or high level API is how much flexibility you would like to have. I would personally stick to low level APIs for "Estimators" part.

5. Low level API - Tensors and variables.

Some of the useful API or functions for tf.Tensor objects:
  • tf.rank(): determines the rank of tf.Tensor object.
  • tf.Tensor slices: the same as python slicing syntax.
  • tf.shape(): determines the shape of tf.Tensor object.
  • tf.reshape(): reshapes the tf.Tensor object.
  • tf.cast(): casting syntax for tf.Tesnor object.
  • tf.eval(): fetching the value of tf.Tensor.
  • tf.Print(): printing the tf.Tensor object.
Some of the useful API or functions for variables:
  • Variable creation: tf.get_variable() recommended. Default type is tf.float32 and can be initialized via tf.glorot_uniform_initializer or tf.zeros_initializer. Input is typically a shape, name, type and initializer.
  • Variable collections: tf.GraphKeys.GLOBAL_VARIABLES (variables that can be shared across multiple devices) and tf.GraphKeys.TRIANABLE_VARIABLES (variables for which TensorFlow will calculate gradients) are default options where each variables are collected. This default can be changed.
  • Device placements: all variables can be manually placed on different devices. Variables must be in the correct devices in distributed settings - tf.train.replica_device_setter automatically place variables in parameter servers.
  • Variable initialization: variables can be initialized with session.run(tf.global_variables_initializer()). If the orders of initializations are important then do not use this but initialize variables individually.
  • Sharing variables: sharing of variables can be done via (1) explictly passing tf.Variable objects around and (2) implicitly wrapping tf.Variable objects within tf.variable_scope objects.
  • Operations: many tensor flow operation functions exist e.g. tf.add(). Always use them.

6. Low level API - Graphs and Sessions.

Some of the useful information about graphs:
  • Naming operations: naming operations helps for visualizing the graphs. Scopes can add a prefix to all operations created in the same context. 
  • Graphs are typically visualized in TensorBoard.
Some of the useful information about sessions:
  • Creating a session and executing oeprations is done with tf.Session.

7. Low level API - Save and Restore.

To save and restore models, the tf.train.Saver class is typically used. The tf.saved_model.simple_save function is one way to build a saved model suitable for serving.

Saving and restoring the variables:
  • Saving variables: tf.train.Saver() is initialized as ops before the session. Then, during the session saver.save() is used to save the session in a specific dir.
  • Restore variables: tf.train.Saver() is again initialized as ops before the session. Then, during the session, saver.restore() is used to restore the previously saved variables.
  • Choosing variables to save and restore: arguments to tf.train.Saver() function with a name and variable.
  • Checkpoint: We can quickly inspect variables in a checkpoint with the inspect_checkpoint library.
Saving and restoring the model:
  • Simple save: tf.saved_model.simple_save() function to save variables, graphs and its metadata. It can be loaded by TensorFlow serving and supports the Predict API. If it doesnt cover your need then use manual builder APIs to create a SaveModel.
  • Manually build a saved model: tf.saved_model.builder.SavedModelBuilder class provides functionality to save multiple MetaGraphDef. Examples for saving metagraphdef for training and inference are given. 
Loading a saved model:
  • Python - tf.saved_model.loader.load(): arguments are the session, the tags for the MetaGraphDef and the location of the saved model. A specific MetaGraphDef will be restored into the supplied session.
  • C++ - LoadSavedModel(): SavedModelBundle class is used with the arguments such as session options, run options, directory, tag and poitner to the SavedModelBundle instance. In general, inference with C++ comes with less overhead and loading the model from python to C++ is therefore incorporated in a accessible way.
  • TensorFlow serving: The saved model can also be loaded as the TensorFlow server binary.
Using saved model with estimators:
  • Specify the output nodes and the corresponding APIs: serving_input_receiver_fn() that accepts inference requests and prepares them fothe model by adding placeholders to the graph and any additional ops needed to convert data from the input format into the feature Tensors. It must return tf.estimator.export.ServingInputReceiver object.
  • Export the model to the SavedModel format: estimator.export_savedmodel() writes a SavedModel into the one containing MetaGraphDef.
  • Serve the model from a local server and request predictions: Request can be made to perform prediction or inference (not so useful for robots).
CLI to inspect and execute a saved model (command line interface):
  • show: this shows a compuation on a MetaGraphDef in a SavedModel .
  • run: this runs a compuation on a MetaGraphDef.
  • overwrite or --outdir option: this can be used to save the outputs.
Structure of a saved model directory:
  • assets: it is a subfolder containing auxiliary files, such as vocabularies. Assets are copied to the SavedModel location and can be read when loading a specific MetaGraphDef.
  • variables: it is subfolder where high level libraries and users can add their own assets that co-exist with the model, but are not loaded by the graph.
  • When one saves a model in SavedModel format, TensorFlow creates a SavedModel directory consisting of a specific subdirectoreis and files.

8. Debugging.


tf_debug is the API to debug TensorFlow. One needs to write sess = tf_debug.LocalCLIDebugWrapperSession(sess) to inspect the graphs internal state and register special filters for tensor vlaues. Fins the frequently used command line tools.

Use https://github.com/MrGemy95/Tensorflow-Project-Template for big projects.

Comments

Post a Comment

Popular posts from this blog

Notes on "Focal Loss for Dense Object Detection" (RetinaNet)

Image
Focal Loss for Dense Object Detection by Lin et al (2017) The central idea of this paper is a proposal for a new loss function to train one-stage detectors which works effectively for class imbalance problems (typically found in one-stage detectors such as SSD). The authors also introduce RetinaNet based on the proposed loss function which outperforms the state-of-the-art detection algorithms including the two-stage detectors. Focal loss function: pt is defined as p if y = 1 and p-1 otherwise. In this way, CE(pt) is the cross entropy where p is the probability score. As shown below, the cross entropy is denoted blue (gamma = 0). By adding the factor (see the focal loss FL(pt)) the loss function computes smaller value for the well classified examples and puts more focus on hard and misclassified eamples. In this way the easy background examples are dealt with, which enables the training of highly accurate desne object detection algorithms. Just like alpha-cross entropy is the comm
Read more

Introduction to Bayesian methods

Image
Introduction to Bayesian methods: 1. Bayesian way of thinking: A man is running. Why? Some of the possible explanations are (1) this man is in hurry, (2) this man is running, (3) this man just spotted a dragon, and (4) this man always runs. There are three principles in Bayesian way of thinking which we can use to conclude which of these explanations are the most reasonable. These principles are: 1. Use the prior knowledge on the problem - there is no dragon. 2. Choose the answer that explains the data the most - the man is not wearing the runner. 3. Avoid making extra assumptions - crude to think that any man always run. The answer number "1. this man is in hurry" should be chosen following this principles. 2. Review of probability: Checklists are: 1. What is the probability? 2. What are random variables? 3. What is the probability mass function (PMF)? 4. What is the probability density function (PDF)? 5. What does it mean for the random variables X and Y
Read more

Conjugate Priors

Conjugate Priors. 1. Analytical inference: See the Bayes formulae. The problem is that the evidence is difficult to estimate for the analytical inference. One way to get away is the maximum posterior (MAP) and is very easy to compute, but it comes with many problems. 1. Lack of the in-variance to re-parameterization. 2. The MAP cannot be used as the prior. 3. The MAP is an untypical point (the mode). 4. The credible regions may not be computed. 2. Conjugate distributions. Another approach to avoid computing the evidence is called the conjugate distributions. As the likelihood and the evidence is fixed by our model, we can only vary the prior so that it is easier to compute the posterior. I f the posterior distribution p(θ|x) are in the same family as the prior distribution p(θ), the prior and posterior are then called conjugate distributions, and the prior is called a conjugate prior for the likelihood. For example, the Gaussian family is conjugate to itself (or self-
Read more