TensorFlow умер. Да здравствует TensorFlow 2.0!


Оглавление (нажмите, чтобы открыть):

TensorFlow – Установка

Для установки TensorFlow в операционной системе Windows выполните следующие шаги.

Шаг 1 – Убедитесь, что версия Python устанавливается.

Шаг 2 – Пользователь может выбрать любой механизм для установки TensorFlow в систему. Мы рекомендуем «Pip» и «Anaconda». Pip – это команда, используемая для выполнения и установки модулей в Python.

Перед установкой TensorFlow нам необходимо установить инфраструктуру Anaconda в нашей системе.

После успешной установки проверьте в командной строке команду «conda».

Шаг 3 – Выполните следующую команду, чтобы инициализировать установку TensorFlow:

Он загружает необходимые пакеты, необходимые для установки TensorFlow.

Шаг 4 – После успешной настройки среды важно активировать модуль TensorFlow.

Шаг 5 – Используйте pip для установки «Tensorflow» в системе. Команда, используемая для установки, указана ниже:

После успешной установки важно знать пример выполнения программы TensorFlow.

Следующий пример помогает нам понять основную программу создания «Привет мир» в TensorFlow.

Код для реализации первой программы упомянут ниже:

Если вы нашли ошибку, пожалуйста, выделите фрагмент текста и нажмите Ctrl+Enter.

Python. TensorFlow для чайников. Часть 1.

TensorFlow — это мощная библиотека для создания нейронных сетей. На уроке Пишем нейросеть на Python. Часть 1 я уже писал, как сделать нейросеть на Python. Поскольку вторая часть пока в работе, решил написать статью про TensorFlow, ибо это поистине великолепный инструмент. Если раньше нам надо было создавать свои классы, чтобы хранить информацию о весах нейрона, объединять их в слои и еще добавлять туда алгоритмы обучения, то теперь все это есть в TensorFlow.

Как его установить? Сначала скачиваем whl файл, затем устанавливаем его, примерно вот так:

Итак, сначала попробуем создать простой нейрон с одним входом и выходом, который реализует линеную функцию:

Мы создали простой нейрон со входом, выходом, и одним входным весовых коэффициентом. Чтобы попользоваться им, нам нужно создать сессию:

Затем инициализировать переменные:

После этого мы можем вывести значение на экран:

У нас выведет 0.8 — результат умножения входного значения 1 на весовой коэффициент 0.8.

Теперь попробуем его обучить. Допустим, мы желаем получить на выходе не 0.8, а к примеру 2. В этом случае нам надо задать желаемое значение, функцию ошибки и оптимизатор:

А потом еще надо итерационно, много раз применить оптимизацию (в цикле), вот так:

А вот весь код нашего примера:

В результате мы получим число, близкое к 2 (к нашему желаемому выходу), например 1.9929.

Часто для оцени процесса обучения смотрят график обучения, на котором, как правило, отображается ошибка обучения (как она уменьшается со временем). Давайте и мы так же сделаем, воспользовавшись питоновской библиотекой научной графики.

Для этого нам надо импортунть саму библиотеку, в цикл обучения вставать код, который будет выводить значение ошибки на каждой итерации в список или массив, и отобразить этот список или массив на графике. Вот полный код программы с данными изменениями:

Эффективный TensorFlow 2.0

В TensorFlow 2.0 сделан ряд изменений делающих пользователей TensorFlow более продуктивными. TensorFlow 2.0 удалил избыточные API, после чего API стали более согласованными (Unified RNNs, Unified Optimizers), и лучше интегрировался с Python runtime, с Eager execution.

Многие RFCs объяснили изменения которые вошли в TensorFlow 2.0. Это руководство представляет взгляд на кто как должна выглядеть разработка в TensorFlow 2.0. Предполагается, что вы знакомы с TensorFlow 1.x.

Короткая выдержка основных изменений

Очистка API

Много API либо удалены либо перемещены в TF 2.0. Некоторые из основных изменений включают удаление tf.app , tf.flags , и tf.logging в пользу absl-py который сейчас с открытым исходным кодом, перенос проектов которые находились в tf.contrib , и очистки основного пространства имен tf.* путем перемещения редко используемых функций в подпакеты наподобие tf.math . Неокторые API были замещены своими эквивалентами 2.0 — tf.summary , tf.keras.metrics , и tf.keras.optimizers . Наиболее простым способом автоматически применить эти переименования является использование скрипта обновления v2.

Eager execution

В TensorFlow 1.X от пользователей требовалось вручную собирать абстрактное синтаксическое дерево (граф) выполняя tf.* API запросы. Затем пользователи должны вручную скомпилировать абстрактное синтаксическое дерево путем передачи множества выходных и входных тензоров в вызов session.run() . TensorFlow 2.0 выполняется сразу же (как это обычно делает Python) и в 2.0, графы и сессии должны рассматриваться как детали реализации.

Одним заметным побочным продуктом eager execution является то, что tf.control_dependencies() более не требуется, так как все строки кода выполняются по очереди (в пределах tf.function , код с побочными эффектами выполняется в том порядке в котором он написан).

Нет больше глобалов

TensorFlow 1.X згачительно зависел от неявных глобальных пространств имен. Когда вы вызывали tf.Variable() , она помещалась в граф по умолчанию, и она оставалась там даже если вы потеряли track переменной Python указывавшей на него. Вы можете затем восстановить ту tf.Variable , но только если вы знали имя с которым она была создана. Это было сложно сделать если вы не контролировали создание переменных. В результате этого, размножались все виды механизмов пытавшиеся помочь пользователям снова найти их переменные, а для фреймворков — найти созданные пользователями переменные: Области переменных, глобальные коллекции, методы помощники такие как tf.get_global_step() , tf.global_variables_initializer() , оптимизаторы неявно вычисляющие градиенты по всем обучаемым переменным, и т.д. TensorFlow 2.0 устраняет все эти механизмы (Variables 2.0 RFC) в пользу механизма по умолчанию: Отслеживайте свои переменные! Если вы потеряли след tf.Variable , он будет очищен сборщиком мусора.

Требование отслеживать мусор создает дополнительную работу для пользователя, но с объектами Keras (см. ниже), нагрузка минимизирована.

Функции, не сессии

Вызов session.run() почти похож на вызов функции: Вы определяете вводные данные, функция вызывается и вы получаете набор результатов. В TensorFlow 2.0, вы можете декорировать функцию Python используя tf.function() чтобы отметить ее для JIT компиляции так что TensorFlow выполняет его как единый граф (Functions 2.0 RFC). Этот механизм позволяет TensorFlow 2.0 получить все преимущества режима графа:

  • Производительность: функция может быть оптимизирована (node pruning, kernel fusion, etc.)
  • Портативность: функция может быть экспортирована / реимпортирована (RFC SavedModel 2.0, что позволяет пользователям повторно использовать и делиться модульными функциями TensorFlow.

Благодаря возможности свободно перемежать код Python и TensorFlow пользователи могут воспользоваться преимуществами выразительности Python. Но переносимый TensorFlow выполняется в контекстах, таких как mobile, C ++ и JavaScript без интерпретатора Python. Чтобы пользователям не нужно было переписывать свой код при добавлении @ tf.function , AutoGraph преобразует подмножество Python конструируя его в эквивалентах TensorFlow:

  • for / while ->tf.while_loop ( break and continue are supported)
  • if ->tf.cond
  • for _ in dataset -> dataset.reduce

AutoGraph поддерживает произвольные вложения control flow, что делает возможным эффективно и кратко реализовать многие сложные программы машинного обучения, такие как реккурентные модели, обучение с подкреплением, пользовательские циклы обучения и многое другое.

Рекомендации характерные для TensorFlow 2.0

Рефакторьте ваш код в меньшие функции

Обычный пользовательский паттерн в TensorFlow 1.X была стратегия «kitchen sink»(кухонная мойка), где предварительно выкладывалось объединение всех возможных вычислений, а потом выбранные тензоры оценивались с session.run() . В TensorFlow 2.0, пользователям необходимо отрефакторить свой код в меньшие функции которые вызываются по мере необходимости. В общем, не обязательно декорировать каждую из этих функций с tf.function ; используйте tf.function только для декорирования высокоуровневых вычислений — например, один шаг обучения или проход вперед в вашей модели.

Используйте слои и модели Keras для управления переменными

Keras models and layers offer the convenient variables and trainable_variables properties, which recursively gather up all dependent variables. This makes it easy to manage variables locally to where they are being used.

Модели и слои Keras предлагают удобные свойства variables и trainable_variables , которые рекурсивно собирают все зависимые переменные. Это облегчает локальное управление переменными в том месте, где они использовались.

с версией Keras:

Слои/модели Keras наследуются от tf.train.Checkpointable и интегрированы с @ tf.function , что позволяет напрямую проверять или экспортировать SavedModels из объектов Keras. Вам не обязательно использовать Keras .fit () API чтобы воспользоваться этими интеграциями.

Вот пример transfer learning, который демонстрирует, как Keras облегчает сбор подмножества релевантных переменных. Допустим, вы обучаете multi-headed model with a shared trunk:

Комбинируйте tf.data.Datasets и @tf.function

При итерации по тренировочным данным, которые помещаются в память, свободно используйте регулярную итерацию Python. Иначе, tf.data.Dataset — лучший способ для передачи тренировочных данных с диска. Данные являются iterables (не iterators), и работают так же, как и другие Python iterables в режиме Eager. Вы можете полностью использовать свойства dataset async prefetching/streaming упаковав свой код в tf.function () , которая заменяет итерацию Python эквивалентным графом операции использующим AutoGraph.

If you use the Keras .fit() API, you won’t have to worry about dataset iteration.

Воспользуйтесь преимуществами AutoGraph с Python control flow

AutoGraph предоставляет способ преобразования зависящего от данных control flow в эквивалентый режим графа, например tf.cond и tf.while_loop .

Одно обычное место, где появляется зависящий от данных control flow находится sequence models. tf.keras.layers.RNN оборачивает ячейку RNN, позволяя вам статически или динамически развернуть recurrence. Например, вы может переопределить динамическую развертку следующим образом:

Для более подробного обзора свойств AutoGraph, смотри руководство.

tf.metrics аггрегирует данные and tf.summary ведет их лог

Для лога summaries используйте tf.summary. (Scalar | histogram | . ) и перенаправьте его на writer используя context manager. (Если вы опустите context manager, ничего случится.) В отличие от TF 1.x, summaries отправляются непосредственно writer; там нет отдельной операции «merge» и отдельного вызова add_summary() , что означает, что значение step должно быть указано на месте вызова.

Чтобы объединить данные перед их записью в виде summaries, используйте tf.metrics . Метрика являются stateful: они накапливают значения и возвращают совокупный результат, когда вы вызовите .result() . Очистите накопленные значения с помощью .reset_states () .

Визуализируйте сгенерированные результаты направив TensorBoard в директорий с summary log:

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Что нового в TensorFlow 2.0

Aug 30 ・4 min read

TensorFlow как никто другой оказывает влияние на machine learning комьюнити. Фреймворк с недавних пор уже в бете с литерой 2.0. Это достаточно серьезный апдейт и в этом тексте хочу осветить наиболее значимые моменты.

Это не руководство по AI и для понимания вы уже должны быть знакомы с TensorFlow.

Установка и демонстрационный набор данных

Используем Jupyter Notebook:


Если необходимо использовать графический процессор (CUDA):

Нам также будет необходим тестовый набор данных. Будем использовать набор данных Adult из архива UCI.

Набор данных представляет собой задачу двоичной классификации, которая заключается в прогнозировании, будет ли человек зарабатывать более 50 000 долларов в год.

Для начала проведем базовую предварительную обработку данных, а затем настроим разделение данных в соотношении 80:20:

Если TensorFlow у нас установился, датасет загрузился, то можно переходить к самому интересному.

1. Eager execution по умолчанию

В TensorFlow 2.0 больше не нужно создавать сессию и запускать в нем граф вычислений. Eager execution включено по умолчанию в версии 2.0, так что вы можете создавать модели и запускать их мгновенно. Отключить eager execution можно следующим образом:

2. tf.function и AutoGraph

В то время как eager execution позволяет вам работать с императивным программированием, но как только речь заходит о распределенном обучении, полномасштабной оптимизации, производственные среды style graph execution TensorFlow 1.x имеет свои преимущества по сравнению с eager execution. В TensorFlow 2.0 вы сохраняете выполнение на основе графов, но более гибким способом. Это можно сделать с помощью tf.function и AutoGraph.

tf.function позволяет определять графы TensorFlow с синтаксисом в стиле Python с помощью функции AutoGraph. AutoGraph поддерживает широкий диапазон совместимости с Python, включая оператор if, цикл for, цикл while, итераторы и т. д. Однако существуют ограничения. Здесь вы можете ознакомиться с полным списком доступных на данный момент операторов. Ниже приведен пример, показывающий, как легко определить граф TensorFlow с помощью простого декоратора.

Обратите внимание, что вам не нужно было создавать сессии или поля для заполнения для запуска функции single_layer (). Это одна из замечательных функций tf.function. Из-за этого он выполняет все необходимые оптимизации, чтобы код работал быстрее.

3. tf.variable_scope больше не нужен

В версии TensorFlow 1.x, для того, чтобы использовать tf.layers в качестве переменных и использовать их повторно, Вам нужно был использовать блок tf.variable. Но этого больше не нужно делать в TensorFlow 2.0. Из-за присутствия keras в качестве центрального высокоуровневого API в TensorFlow 2.0 все слои, созданные с использованием tf.layers, могут быть легко помещены в определение tf.keras.Sequential. Это значительно облегчает чтение кода, а также позволяет отслеживать переменные и потери.

В приведенном выше примере вы передали набор данных для обучения через model просто для получения необработанных выходных вероятностей. Обратите внимание, что это просто прямой проход. Вы можете, конечно, пойти вперед и обучать свою модель:

Вы можете получить список обучаемых параметров модели по слоям, например:

4. Кастомные слои это просто

В исследованиях в области машинного обучения и в промышленных приложениях часто возникает необходимость в написании кастомных слоев для удовлетворения конкретных случаев использования. TensorFlow 2.0 позволяет очень просто написать собственный слой и использовать его вместе с существующими слоями. Вы также можете настроить прямой проход вашей модели любым удобным для вас способом.

Чтобы создать кастомный слой, проще всего расширить класс Layer из tf.keras.layers, а затем определить его соответствующим образом. Вы создадите кастомный слой, а затем определите его прямые вычисления.

Можно создать несколько слоев, расширяя класс Model из tf.keras. Подробнее о создании моделей можно узнать здесь.

5. Гибкость в обучении модели

TensorFlow может использовать автоматическое дифференцирование для вычисления градиентов функции потерь относительно параметров модели. tf.GradientTape создает ленту в контексте, который используется TensorFlow для отслеживания градиентов, записанных при каждом вычислении на этой ленте. Чтобы понять это, давайте определим модель на более низком уровне, расширив класс tf.keras.Model.

Обратите внимание, что топология этой модели точно такая же, как и та, которую вы определили ранее. Чтобы обучить эту модель с использованием автоматического дифференцирования, вам нужно по-разному определить функцию потерь и оптимайзер:

Теперь вы определите метрики, которые будут использоваться для измерения производительности сети, включающей ее обучение. Под производительностью здесь подразумеваются потеря и точность модели.

tf.data предоставляет служебные методы для определения конвейеров входных данных. Это особенно полезно, когда вы имеете дело с большим объемом данных.

Теперь вы определите генератор данных, который будет генерировать пакеты данных во время обучения модели.

Теперь вы готовы обучать модель, используя tf.GradientTape. Во-первых, вы определите метод, который будет тренировать модель с данными, которые вы только что определили, используя tf.data.DataSet. Вы также оберните этапы обучения модели с помощью декоратора tf.function, чтобы ускорить вычисления.

6. Наборы данных TensorFlow

Отдельный модуль DataSets используется для отточенной работы с сетевой моделью. Вы уже видели это в предыдущем примере. В этом разделе вы увидите, как можно загрузить набор данных MNIST именно так, как вам нужно.

Вы можете установить библиотеку tenorflow_datasets с помощью pip. Как только он установлен, вы готовы приступать к работе. Он предоставляет несколько вспомогательных функций, которые помогут вам гибко подготовить конвейер построения набора данных. Вы можете узнать больше об этих функциях здесь и здесь. Теперь вы увидите, как вы можете построить конвейер ввода данных для загрузки в наборе данных MNIST.

Вы можете игнорировать предупреждение. Обратите внимание, как оригинально tennsflow_datasets обрабатывает конвейер.

Теперь вы можете отобразить первое изображение из коллекции изображений, в которую вы загрузили. Обратите внимание, что tennsflow_datasets работает в активном режиме и настроен на основе графа.

7. Политика автоматической смешанной точности

Политика смешанной точности была предложена NVIDIA в прошлом году. Оригинал статьи находится здесь. Кратко об основной идеи политики смешанной точности: использовать смесь половин (FP16) и полной точности (FP32) и использовать преимущества обоих миров. Она показала потрясающие результаты в обучении очень глубоких нейронных сетей (как с точки зрения времени и оценки).

Если вы работаете в среде графического процессора с поддержкой CUDA (например, Volta Generation, Tesla T4) и установили вариант TensorFlow 2.0 с графическим процессором, вы можете дать TensorFlow команду обучаться со смешанной точностью, например, так:

Это автоматически распределит операции графа TensorFlow соответственно. Вы сможете увидеть хороший скачок производительности вашей модели. Вы также можете оптимизировать основные операции TensorFlow с помощью политики смешанной точности. Подробнее почитайте в этой статье.

8. Распределенное обучение

TensorFlow 2.0 позволяет очень легко распределять процесс обучения по нескольким графическим процессорам. Это особенно полезно для производственных целей, когда вам приходится встречать сверхтяжелые нагрузки. Это так же просто, как поместить ваш тренировочный блок в блок with.

Сначала вы указываете стратегию распределения следующим образом:

Зеркальная стратегия создает одну реплику для каждого графического процессора, а переменные модели одинаково отражаются в разных графических процессорах. Теперь вы можете использовать определенную стратегию следующим образом:

Обратите внимание, что приведенный выше фрагмент кода будет полезен только в том случае, если в одной системе настроено несколько графических процессоров. Существует несколько стратегий распределения, которые вы можете настроить. Подробнее об этом можно прочитать здесь.

9. TensorBoard в Jupyter notebook

Это, пожалуй, самая захватывающая часть этого апдейта. Вы можете визуализировать обучение модели непосредственно в Jupyter notebook через TensorBoard. Новый TensorBoard имеет множество интересных функций, таких как профилирование памяти, просмотр данных изображения, включая матрицу неточностей, граф концептуальной модели и так далее. Подробнее об этом можно прочитать здесь.

В этом разделе вы сконфигурируете свою среду так, чтобы TensorBoard отображался в Jupyter. Сначала вам нужно загрузить расширение блокнота tenorboard.notebook:

Теперь вы определите обратный вызов TensorBoard с помощью модуля tf.keras.callbacks.

Перестройте модель, используя Sequential API tf.keras:

Обучение и тестовые наборы были модифицированы для различных целей. Так что будет хорошо, если разделить их еще раз:

Вы все готовы обучать модель:

Панель инструментов TensorBoard должна быть загружена в Jupyter, и вы сможете отслеживать метрику обучения и валидац

Обзор TensorFlow 2.0

На саммите разработчиков TensorFlow Dev Summit 2020, который состоялся 6–7 марта в Калифорнии, Google представила TensorFlow 2.0, и кажется, там всё по-другому.

Поскольку TensorFlow 1.X. был не самым простым в использовании софтом, разработчики сосредоточились на том, чтобы максимально увеличить юзабилити TF2 и сделать юзер френдли интерфейс, с API как для начинающих, так и для экспертов создания моделей машинного обучения.

В TensorFlow 2.0 удалили избыточные API, а оставшиеся стали более согласованными (Unified RNNs, Unified Optimizer). Переработали API для Keras — инструмента с открытым исходным кодом, который облегчит использование сред разработки AI. Также были улучшены интеграция со средой Python и режим Eager execution.

Читайте также

Несмотря на все новые фичи и улучшения, большинство из нас ждали релиза двух дополнительных модулей, приуроченного Google к выходу новой версии TF2. Они предназначены для того, чтобы помочь разработчикам встроить средства контроля конфиденциальности непосредственно в софт AI для защиты персональной информации пользователей.

Первый модуль — TensorFlow Privacy, он позволяет алгоритмам МО фильтровать конфиденциальные пользовательские данные и не обрабатывать их. Это достигается за счёт автоматического отсеивания входящих данных, отличных от тех, которые обычно принимает алгоритм.

Второй новый модуль, представленный Google, называется TensorFlow Federated. Это ПО нацелено на использующие ИИ мобильные сервисы, число которых постоянно растёт.

Из-за ограниченной вычислительной мощности современных мобильных девайсов приложения обычно используют облачные серверы для обработки пользовательской информации и её анализа. TensorFlow Federated позволяет приложениям выполнять анализ непосредственно на пользовательском устройстве. Разработчики могут обучать и улучшать свои алгоритмы AI, используя агрегированные пользовательские данные без непосредственного к ним доступа, что повышает безопасность пользователей.

Как и TensorFlow, оба новых модуля имеют открытый исходный код.

  • Более подробно обо всех фичах можно почитать тут: Whats coming in TensorFlow 2.0
  • Видео с презентации:
    Introducing TensorFlow 2.0 and its high-level APIs (TF Dev Summit ’19)
  • Почитать и попробовать альфу тут: Test Drive TensorFlow 2.0 Alpha
  • Если вы ничего не поняли из этой статьи, тут можно записаться на нашу специализацию Data Science

What’s coming in TensorFlow 2.0

Posted by the TensorFlow Team

Thanks to an incredible and diverse community, TensorFlow has grown to become one of the most loved and widely adopted ML platforms in the world. This community includes:

  • Researchers (e.g., to forecast earthquake aftershocks and detect breast cancer).
  • Developers (e.g., to build apps to >In November, TensorFlow celebrated its 3rd birthday with a look back at the features added throughout the years. We’re excited about another major milestone, TensorFlow 2.0.

TensorFlow 2.0 will focus on simplicity and ease of use, featuring updates like:

  • Easy model building with Keras and eager execution.
  • Robust model deployment in production on any platform.
  • Powerful experimentation for research.
  • Simplifying the API by cleaning up deprecated APIs and reducing duplication.

Over the last few years, we’ve added a number of components to TensorFlow. With TensorFlow 2.0, these will be packaged together into a comprehensive platform that supports machine learning workflows from training through deployment. Let’s take a look at the new architecture of TensorFlow 2.0 using a simplified, conceptual diagram as shown below:

Easy model building

In a recent blog post we announced that Keras, a user-friendly API standard for machine learning, will be the central high-level API used to build and train models. The Keras API makes it easy to get started with TensorFlow. Importantly, Keras provides several model-building APIs (Sequential, Functional, and Subclassing), so you can choose the right level of abstraction for your project. TensorFlow’s implementation contains enhancements including eager execution, for immediate iteration and intuitive debugging, and tf.data, for building scalable input pipelines.

Here’s an example workflow (in the coming months, we will be working to update the guides linked below):

  1. Load your data usingtf.data . Training data is read using input pipelines which are created using tf.data . Feature characteristics, for example bucketing and feature crosses are described using tf.feature_column . Convenient input from in-memory data (for example, NumPy) is also supported.
  2. Build, train and val >tf.keras , or usePremade Estimators. Keras integrates tightly with the rest of TensorFlow so you can access TensorFlow’s features whenever you want. A set of standard packaged models (for example, linear or logistic regression, gradient boosted trees, random forests) are also available to use directly (implemented using the tf.estimator API). If you’re not looking to train a model from scratch, you’ll soon be able to use transfer learning to train a Keras or Estimator model using modules from TensorFlow Hub.
  3. Run and debug witheager execution , then usetf.function for the benefits of graphs. TensorFlow 2.0 runs with eager execution by default for ease of use and smooth debugging. Additionally, the tf.function annotation transparently translates your Python programs into TensorFlow graphs. This process retains all the advantages of 1.x TensorFlow graph-based execution: Performance optimizations, remote execution and the ability to serialize, export and deploy easily, while adding the flexibility and ease of use of expressing programs in simple Python.
  4. Use Distribution Strategies for distributed training. For large ML training tasks, the Distribution Strategy API makes it easy to distribute and train models on different hardware configurations without changing the model definition. Since TensorFlow prov >Export to SavedModel. TensorFlow will standardize on SavedModel as an interchange format for TensorFlow Serving, TensorFlow Lite, TensorFlow.js, TensorFlow Hub, and more.

Robust model deployment in production on any platform

TensorFlow has always provided a direct path to production. Whether it’s on servers, edge devices, or the web, TensorFlow lets you train and deploy your model easily, no matter what language or platform you use. In TensorFlow 2.0, we’re improving compatibility and parity across platforms and components by standardizing exchange formats and aligning APIs.

Once you’ve trained and saved your model, you can execute it directly in your application or serve it using one of the deployment libraries:


Powerful experimentation for research

TensorFlow makes it easy to take new ideas from concept to code, and from model to publication. TensorFlow 2.0 incorporates a number of features that enables the definition and training of state of the art models without sacrificing speed or performance:

  • Keras Functional API and Model Sub >tf.GradientTape and tf.custom_gradient .
  • And for even more flexibility and control, the low-level TensorFlow API is always available and working in conjunction with the higher level abstractions for fully customizable logic.

TensorFlow 2.0 brings several new additions that allow researchers and advanced users to experiment, using rich extensions like Ragged Tensors, TensorFlow Probability, Tensor2Tensor, and more to be announced.

Along with these capabilities, TensorFlow provides eager execution for easy prototyping & debugging, Distribution Strategy API and AutoGraph to train at scale, and support for TPUs, making TensorFlow 2.0 an easy to use, customizable, and highly scalable platform for conducting state of the art ML research and translating that research into production pipelines.

Differences between TensorFlow 1.x and 2.0

There have been a number of versions and API iterations since we first open-sourced TensorFlow. With the rapid evolution of ML, the platform has grown enormously and now supports a diverse mix of users with a diverse mix of needs. With TensorFlow 2.0, we have an opportunity to clean up and modularize the platform based on semantic versioning.

Here are some of the larger changes coming:

Additionally, tf.contrib will be removed from the core TensorFlow repository and build process. TensorFlow’s contrib module has grown beyond what can be maintained and supported in a single repository. Larger projects are better maintained separately, while smaller extensions will graduate to the core TensorFlow code. A special interest group (SIG) has been formed to maintain and further develop some of the more important contrib projects going forward. Please engage with this RFC if you are interested in contributing.

Compatibility and Continuity

To simplify the migration to TensorFlow 2.0, there will be a conversion tool which updates TensorFlow 1.x Python code to use TensorFlow 2.0 compatible APIs, or flags cases where code cannot be converted automatically.

Not all changes can be made completely automatically. For example, some deprecated APIs do not have a direct equivalent. That’s why we introduced the tensorflow.compat.v1 compatibility module, which retains support for the full TensorFlow 1.x API (excluding tf.contrib). This module will be maintained through the lifetime of TensorFlow 2.x and will allow code written with TensorFlow 1.x to remain functional.

Additionally, SavedModels or stored GraphDefs will be backwards compatible. SavedModels saved with 1.x will continue to load and execute in 2.x. However, the changes in 2.0 will mean that variable names in raw checkpoints may change, so using a pre-2.0 checkpoint with code that has been converted to 2.0 is not guaranteed to work. See the Effective TensorFlow 2.0 guide for more details.

We believe TensorFlow 2.0 will bring great benefits to the community, and we have invested significant efforts to make the conversion as easy as possible. However, we also recognize that migrating current pipelines will take time, and we deeply care about the current investment the community has made learning and using TensorFlow. We will provide 12 months of security patches to the last 1.x release, in order to give our existing users ample time to transition and get all the benefits of TensorFlow 2.0.

Timeline for TensorFlow 2.0

TensorFlow 2.0 will be available as a public preview early this year. But why wait? You can already develop the TensorFlow 2.0 way by using tf.keras and eager execution, pre-packaged models and the deployment libraries. The Distribution Strategy API is also already partly available today.

We’re very excited about TensorFlow 2.0 and the changes to come. TensorFlow has grown from a software library for deep learning to an entire ecosystem for all types of ML. TensorFlow 2.0 will be simple and easy to use for all users on all platforms.

Please consider joining the TensorFlow community to stay up-to-date and help make machine learning accessible to everyone!

tensorflow 2.0.0

pip install tensorflow Copy PIP instructions

Last released: Sep 30, 2020

TensorFlow is an open source machine learning framework for everyone.

Navigation

Project links

Statistics

View statistics for this project via Libraries.io, or by using Google BigQuery

License: Apache Software License (Apache 2.0)

Tags tensorflow, tensor, machine, learning

Maintainers

Classifiers

  • Development Status
    • 5 — Production/Stable
  • Intended Audience
    • Developers
    • Education
    • Science/Research
  • License
    • OSI Approved :: Apache Software License
  • Programming Language
    • Python :: 2
    • Python :: 2.7
    • Python :: 3
    • Python :: 3.4
    • Python :: 3.5
    • Python :: 3.6
    • Python :: 3.7
  • Topic
    • Scientific/Engineering
    • Scientific/Engineering :: Artificial Intelligence
    • Scientific/Engineering :: Mathematics
    • Software Development
    • Software Development :: Libraries
    • Software Development :: Libraries :: Python Modules

Project description

TensorFlow is an open source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices.

Originally developed by researchers and engineers from the Google Brain team within Google’s AI organization, it comes with strong support for machine learning and deep learning and the flexible numerical computation core is used across many other scientific domains.

Project details

Project links

Statistics

View statistics for this project via Libraries.io, or by using Google BigQuery

License: Apache Software License (Apache 2.0)

Tags tensorflow, tensor, machine, learning

Maintainers

Classifiers

  • Development Status
    • 5 — Production/Stable
  • Intended Audience
    • Developers
    • Education
    • Science/Research
  • License
    • OSI Approved :: Apache Software License
  • Programming Language
    • Python :: 2
    • Python :: 2.7
    • Python :: 3
    • Python :: 3.4
    • Python :: 3.5
    • Python :: 3.6
    • Python :: 3.7
  • Topic
    • Scientific/Engineering
    • Scientific/Engineering :: Artificial Intelligence
    • Scientific/Engineering :: Mathematics
    • Software Development
    • Software Development :: Libraries
    • Software Development :: Libraries :: Python Modules

Release history Release notifications

Download files

Download the file for your platform. If you’re not sure which to choose, learn more about installing packages.

Release 1.15.0

This is the last 1.x release for TensorFlow. We do not expect to update the 1.x branch with features, although we will issue patch releases to fix vulnerabilities for at least one year.

Major Features and Improvements

  • As announced, tensorflow pip package will by default include GPU support (same as tensorflow-gpu now) for the platforms we currently have GPU support (Linux and Windows). It will work on machines with and without Nvidia GPUs. tensorflow-gpu will still be available, and CPU-only packages can be downloaded at tensorflow-cpu for users who are concerned about package size.
  • TensorFlow 1.15 contains a complete implementation of the 2.0 API in its compat.v2 module. It contains a copy of the 1.15 main module (without contrib ) in the compat.v1 module. TensorFlow 1.15 is able to emulate 2.0 behavior using the enable_v2_behavior() function.
    This enables writing forward compatible code: by explicitly importing either tensorflow.compat.v1 or tensorflow.compat.v2 , you can ensure that your code works without modifications against an installation of 1.15 or 2.0.
  • EagerTensor now supports numpy buffer interface for tensors.

  • Add toggles tf.enable_control_flow_v2() and tf.disable_control_flow_v2() for enabling/disabling v2 control flow.
  • Enable v2 control flow as part of tf.enable_v2_behavior() and TF2_BEHAVIOR=1 .
  • AutoGraph translates Python control flow into TensorFlow expressions, allowing users to write regular Python inside tf.function -decorated functions. AutoGraph is also applied in functions used with tf.data , tf.distribute and tf.keras APIS.
  • Adds enable_tensor_equality() , which switches the behavior such that:
    • Tensors are no longer hashable.
    • Tensors can be compared with == and != , yielding a Boolean Tensor with element-wise comparison results. This will be the default behavior in 2.0.
  • Auto Mixed-Precision graph optimizer simplifies converting models to float16 for acceleration on Volta and Turing Tensor Cores. This feature can be enabled by wrapping an optimizer class with tf.train.experimental.enable_mixed_precision_graph_rewrite() .
  • Add environment variable TF_CUDNN_DETERMINISTIC . Setting to «true» or «1» forces the selection of deterministic cuDNN convolution and max-pooling algorithms. When this is enabled, the algorithm selection procedure itself is also deterministic.
  • TensorRT
    • Migrate TensorRT conversion sources from contrib to compiler directory in preparation for TF 2.0.
    • Add additional, user friendly TrtGraphConverter API for TensorRT conversion.
    • Expand support for TensorFlow operators in TensorRT conversion (e.g.
      Gather , Slice , Pack , Unpack , ArgMin , ArgMax , DepthSpaceShuffle ).
    • Support TensorFlow operator CombinedNonMaxSuppression in TensorRT conversion which
      significantly accelerates object detection models.

Breaking Changes

  • Tensorflow code now produces 2 different pip packages: tensorflow_core containing all the code (in the future it will contain only the private implementation) and tensorflow which is a virtual pip package doing forwarding to tensorflow_core (and in the future will contain only the public API of tensorflow). We don’t expect this to be breaking, unless you were importing directly from the implementation.
  • TensorFlow 1.15 is built using devtoolset7 (GCC7) on Ubuntu 16. This may lead to ABI incompatibilities with extensions built against earlier versions of TensorFlow.
  • Deprecated the use of constraint= and .constraint with ResourceVariable.
  • tf.keras :
    • OMP_NUM_THREADS is no longer used by the default Keras config. To configure the number of threads, use tf.config.threading APIs.
    • tf.keras.model.save_model and model.save now defaults to saving a TensorFlow SavedModel.
    • keras.backend.resize_images (and consequently, keras.layers.Upsampling2D ) behavior has changed, a bug in the resizing implementation was fixed.
    • Layers now default to float32 , and automatically cast their inputs to the layer’s dtype. If you had a model that used float64 , it will probably silently use float32 in TensorFlow2, and a warning will be issued that starts with Layer «layer-name» is casting an input tensor from dtype float64 to the layer’s dtype of float32. To fix, either set the default dtype to float64 with tf.keras.backend.set_floatx(‘float64′) , or pass dtype=’float64’ to each of the Layer constructors. See tf.keras.layers.Layer for more information.
    • Some tf.assert_* methods now raise assertions at operation creation time (i.e. when this Python line executes) if the input tensors’ values are known at that time, not during the session.run(). When this happens, a noop is returned and the input tensors are marked non-feedable. In other words, if they are used as keys in feed_dict argument to session.run() , an error will be raised. Also, because some assert ops don’t make it into the graph, the graph structure changes. A different graph can result in different per-op random seeds when they are not given explicitly (most often).

Bug Fixes and Other Changes

  • tf.estimator :
    • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format, which allows the saved checkpoints to be compatible with model.load_weights .
    • Fix tests in canned estimators.
    • Expose Head as public API.
    • Fixes critical bugs that help with DenseFeatures usability in TF2
  • tf.data :
    • Promoting unbatch from experimental to core API.
    • Adding support for datasets as inputs to from_tensors and from_tensor_slices and batching and unbatching of nested datasets.
  • tf.keras :
    • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format, which allows the saved checkpoints to be compatible with model.load_weights .
    • Saving a Keras Model using tf.saved_model.save now saves the list of variables, trainable variables, regularization losses, and the call function.
    • Deprecated tf.keras.experimental.export_saved_model and tf.keras.experimental.function . Please use tf.keras.models.save_model(. save_format=’tf’) and tf.keras.models.load_model instead.
    • Add an implementation=3 mode for tf.keras.layers.LocallyConnected2D and tf.keras.layers.LocallyConnected1D layers using tf.SparseTensor to store weights, allowing a dramatic speedup for large sparse models.
    • Enable the Keras compile API experimental_run_tf_function flag by default. This flag enables single training/eval/predict execution path. With this 1. All input types are converted to Dataset . 2. When distribution strategy is not specified this goes through the no-op distribution strategy path. 3. Execution is wrapped in tf.function unless run_eagerly=True is set in compile.
    • Raise error if batch_size argument is used when input is dataset/generator/keras sequence.
  • tf.lite
    • Add GATHER support to NN API delegate.
    • tflite object detection script has a debug mode.
    • Add delegate support for QUANTIZE .
    • Added evaluation script for COCO minival.
    • Add delegate support for QUANTIZED_16BIT_LSTM .
    • Converts hardswish subgraphs into atomic ops.
  • Add support for defaulting the value of cycle_length argument of tf.data.Dataset.interleave to the number of schedulable CPU cores.
  • parallel_for : Add converter for MatrixDiag .
  • Add narrow_range attribute to QuantizeAndDequantizeV2 and V3.
  • Added new op: tf.strings.unsorted_segment_join .
  • Add HW acceleration support for topK_v2 .
  • Add new TypeSpec classes.
  • CloudBigtable version updated to v0.10.0.
  • Expose Head as public API.
  • Update docstring for gather to properly describe the non-empty batch_dims case.
  • Added tf.sparse.from_dense utility function.
  • Improved ragged tensor support in TensorFlowTestCase .
  • Makes the a-normal form transformation in Pyct configurable as to which nodes are converted to variables and which are not.
  • ResizeInputTensor now works for all delegates.
  • Add EXPAND_DIMS support to NN API delegate TEST: expand_dims_test
  • tf.cond emits a StatelessIf op if the branch functions are stateless and do not touch any resources.
  • tf.cond , tf.while and if and while in AutoGraph now accept a nonscalar predicate if has a single element. This does not affect non-V2 control flow.
  • tf.while_loop emits a StatelessWhile op if the cond and body functions are stateless and do not touch any resources.
  • Refactors code in Quant8 LSTM support to reduce TFLite binary size.
  • Add support of local soft device placement for eager op.
  • Add HW acceleration support for LogSoftMax .
  • Added a function nested_value_rowids for ragged tensors.
  • Add guard to avo >tf.math.cumulative_logsumexp operation .
  • Add tf.ragged.stack .
  • Fix memory allocation problem when calling AddNewInputConstantTensor .
  • Delegate application failure leaves interpreter in valid state.
  • Add check for correct memory alignment to MemoryAllocation::MemoryAllocation() .
  • Extracts NNAPIDelegateKernel from nnapi_delegate.cc
  • Added support for FusedBatchNormV3 in converter.
  • A ragged to dense op for directly calculating tensors.
  • Fix accidental quadratic graph construction cost in graph-mode tf.gradients() .
  • The precision_mode argument to TrtGraphConverter is now case insensitive.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

a6802739, Aaron Ma, Abdullah Selek, Abolfazl Shahbazi, Ag Ramesh, Albert Z. Guo, Albin Joy, Alex Itkes, Alex Sergeev, Alexander Pivovarov, Alexey Romanov, alhkad, Amit Srivastava, amoitra, Andrew Lihonosov, Andrii Prymostka, Anuj Rawat, Astropeak, Ayush Agrawal, Bairen Yi, Bas Aarts, Bastian Eichenberger, Ben Barsdell, Benjamin Peterson, bhack, Bharat Raghunathan, Bhavani Subramanian, Bryan Cutler, candy.dc, Cao Zongyan, Captain-Pool, Casper Da Costa-Luis, Chen Guoyin, Cheng Chang, chengchingwen, Chong Yan, Choong Yin Thong, Christopher Yeh, Clayne Robison, Coady, Patrick, Dan Ganea, David Norman, Denis Khalikov, Deven Desai, Diego Caballero, Duncan Dean, Duncan Riach, Dwight J Lyle, Eamon Ito-Fisher, eashtian3, EFanZh, ejot, Elroy Ashtian Jr, Eric Schweitz, Fangjun Kuang, Fei Hu, fo40225, formath, Fred Reiss, Frederic Bastien, Fredrik Knutsson, G. Hussain Chinoy, Gabriel, gehring, George Grzegorz Pawelczak, Gianluca Varisco, Gleb Popov, Greg Peatfield, Guillaume Klein, Gurpreet Singh, Gustavo Lima Chaves, haison, Haraldur TóMas HallgríMsson, HarikrishnanBalagopal, HåKon Sandsmark, I-Hong, Ilham Firdausi Putra, Imran Salam, Jason Zaman, Jason Zavaglia, jayhpark530, jefby, Jeff Daily, Jeffrey Poznanovic, Jekyll Lai, Jeroen BéDorf, Jerry Shih, jerryyin, jiakai, JiangXIAO, Joe Bowser, Joel Shapiro, Johan Gunnarsson, Jojimon Varghese, Joon, Josh Beal, Julian Niedermeier, Jun Wan, Junqin Zhang, Junyuan Xie, Justin Tunis, Kaixi Hou, Karl Lessard, Karthik Muthuraman, Kbhute-Ibm, khanhlvg, Koock Yoon, kstuedem, Kyuwon Kim, Lakshay Tokas, leike666666, leonard951, Leslie-Fang, Leslie-Fang-Intel, Li, Guizi, Lukas Folle, Lukas Geiger, Mahmoud Abuzaina, Manraj Singh Grover, Margaret Maynard-Reid, Mark Ryan, Matt Conley, Matthew Bentham, Matthew Denton, mbhuiyan, mdfaijul, Mei Jie, merturl, MichaelKonobeev, Michal W. Tarnowski, minds, mpppk, musikisomorphie, Nagy Mostafa, Nayana Thorat, Neil, Niels Ole Salscheider, Niklas SilfverströM, Niranjan Hasabnis, ocjosen, olramde, Pariksheet Pinjari, Patrick J. Lopresti, Patrik Gustavsson, per1234, PeterLee, Phan Van Nguyen Duc, Phillip Kravtsov, Pooya Davoodi, Pranav Marathe, Putra Manggala, Qingqing Cao, Rajeshwar Reddy T, Ramon ViñAs, Rasmus Diederichsen, Reuben Morais, richardbrks, robert, RonLek, Ryan Jiang, saishruthi, Saket Khandelwal, Saleem Abdulrasool, Sami Kama, Sana-Damani, Sergii Khomenko, Severen Redwood, Shubham Goyal, Sigrid Keydana, Siju Samuel, sleighsoft, smilu97, Son Tran, Srini511, srinivasan.narayanamoorthy, Sumesh Udayakumaran, Sungmann Cho, Tae-Hwan Jung, Taehoon Lee, Takeshi Watanabe, TengLu, terryky, TheMindVirus, ThisIsIsaac, Till Hoffmann, Timothy Liu, Tomer Gafner, Tongxuan Liu, Trent Lo, Trevor Morris, Uday Bondhugula, Vasileios Lioutas, vbvg2008, Vishnuvardhan Janapati, Vivek Suryamurthy, Wei Wang, Wen-Heng (Jack) Chung, wenxizhu, William D. Irons, winstonq, wyzhao, Xiaoming (Jason) Cui, Xinan Jiang, Xinping Wang, Yann-Yy, Yasir Modak, Yong Tang, Yongfeng Gu, Yuchen Ying, Yuxin Wu, zyeric, 王振华 (Zhenhua Wang)

goldiegadde released this Oct 8, 2020 · 22 commits to r1.15 since this release

Release 1.15.0-rc3

This is the last 1.x release for TensorFlow. We do not expect to update the 1.x branch with features, although we will issue patch releases to fix vulnerabilities for at least one year.

Major Features and Improvements

  • As announced, tensorflow pip package will by default include GPU support (same as tensorflow-gpu now) for the platforms we currently have GPU support (Linux and Windows). It will work on machines with and without Nvidia GPUs. tensorflow-gpu will still be available, and CPU-only packages can be downloaded at tensorflow-cpu for users who are concerned about package size.
  • TensorFlow 1.15 contains a complete implementation of the 2.0 API in its compat.v2 module . It contains a copy of the 1.15 main module (without contrib ) in the compat.v1 module . TensorFlow 1.15 is able to emulate 2.0 behavior using the enable_v2_behavior() function.
    This enables writing forward compatible code: by explicitly importing either tensorflow.compat.v1 or tensorflow.compat.v2 , you can ensure that your code works without modifications against an installation of 1.15 or 2.0.
  • EagerTensor now supports numpy buffer interface for tensors.
  • Add toggles tf.enable_control_flow_v2() and tf.disable_control_flow_v2() for enabling/disabling v2 control flow.
  • Enable v2 control flow as part of tf.enable_v2_behavior() and TF2_BEHAVIOR=1 .
  • AutoGraph translates Python control flow into TensorFlow expressions, allowing users to write regular Python inside tf.function -decorated functions. AutoGraph is also applied in functions used with tf.data , tf.distribute and tf.keras APIS.
  • Adds enable_tensor_equality() , which switches the behavior such that:
    • Tensors are no longer hashable.
    • Tensors can be compared with == and !=, yielding a Boolean Tensor with element-wise comparison results. This will be the default behavior in 2.0.

  • Auto Mixed-Precision graph optimizer simplifies converting models to float16 for acceleration on Volta and Turing Tensor Cores. This feature can be enabled by wrapping an optimizer class with tf.train.experimental.enable_mixed_precision_graph_rewrite() .
  • Add environment variable TF_CUDNN_DETERMINISTIC . Setting to «true» or «1» forces the selection of deterministic cuDNN convolution and max-pooling algorithms. When this is enabled, the algorithm selection procedure itself is also deterministic.
  • TensorRT
    • Migrate TensorRT conversion sources from contrib to compiler directory in preparation for TF 2.0.
    • Add additional, user friendly TrtGraphConverter API for TensorRT conversion.
    • Expand support for TensorFlow operators in TensorRT conversion (e.g.
      Gather , Slice , Pack , Unpack , ArgMin , ArgMax , DepthSpaceShuffle ).
    • Support TensorFlow operator CombinedNonMaxSuppression in TensorRT conversion which
      significantly accelerates object detection models.

Breaking Changes

  • Tensorflow code now produces 2 different pip packages: tensorflow_core containing all the code (in the future it will contain only the private implementation) and tensorflow which is a virtual pip package doing forwarding to tensorflow_core (and in the future will contain only the public API of tensorflow). We don’t expect this to be breaking, unless you were importing directly from the implementation.
  • TensorFlow 1.15 is built using devtoolset7 (GCC7) on Ubuntu 16. This may lead to ABI incompatibilities with extensions built against earlier versions of TensorFlow.
  • Deprecated the use of constraint= and .constraint with ResourceVariable.
  • tf.keras :
    • OMP_NUM_THREADS is no longer used by the default Keras config. To configure the number of threads, use tf.config.threading APIs.
    • tf.keras.model.save_model and model.save now defaults to saving a TensorFlow SavedModel.
    • keras.backend.resize_images (and consequently, keras.layers.Upsampling2D ) behavior has changed, a bug in the resizing implementation was fixed.
    • Layers now default to float32 , and automatically cast their inputs to the layer’s dtype. If you had a model that used float64 , it will probably silently use float32 in TensorFlow2, and a warning will be issued that starts with Layer «layer-name» is casting an input tensor from dtype float64 to the layer’s dtype of float32. To fix, either set the default dtype to float64 with tf.keras.backend.set_floatx(‘float64′) , or pass dtype=’float64’ to each of the Layer constructors. See tf.keras.layers.Layer for more information.
    • Some tf.assert_* methods now raise assertions at operation creation time (i.e. when this Python line executes) if the input tensors’ values are known at that time, not during the session.run(). When this happens, a noop is returned and the input tensors are marked non-feedable. In other words, if they are used as keys in feed_dict argument to session.run() , an error will be raised. Also, because some assert ops don’t make it into the graph, the graph structure changes. A different graph can result in different per-op random seeds when they are not given explicitly (most often).

Bug Fixes and Other Changes

  • tf.estimator :
    • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format, which allows the saved checkpoints to be compatible with model.load_weights .
    • Fix tests in canned estimators.
    • Expose Head as public API.
    • Fixes critical bugs that help with DenseFeatures usability in TF2
  • tf.data :
    • Promoting unbatch from experimental to core API.
    • Adding support for datasets as inputs to from_tensors and from_tensor_slices and batching and unbatching of nested datasets.
  • tf.keras :
    • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format, which allows the saved checkpoints to be compatible with model.load_weights .
    • Saving a Keras Model using tf.saved_model.save now saves the list of variables, trainable variables, regularization losses, and the call function.
    • Deprecated tf.keras.experimental.export_saved_model and tf.keras.experimental.function . Please use tf.keras.models.save_model(. save_format=’tf’) and tf.keras.models.load_model instead.
    • Add an implementation=3 mode for tf.keras.layers.LocallyConnected2D and tf.keras.layers.LocallyConnected1D layers using tf.SparseTensor to store weights, allowing a dramatic speedup for large sparse models.
    • Enable the Keras compile API experimental_run_tf_function flag by default. This flag enables single training/eval/predict execution path. With this 1. All input types are converted to Dataset . 2. When distribution strategy is not specified this goes through the no-op distribution strategy path. 3. Execution is wrapped in tf.function unless run_eagerly=True is set in compile.
    • Raise error if batch_size argument is used when input is dataset/generator/keras sequence.
  • tf.lite
    • Add GATHER support to NN API delegate.
    • tflite object detection script has a debug mode.
    • Add delegate support for QUANTIZE .
    • Added evaluation script for COCO minival.
    • Add delegate support for QUANTIZED_16BIT_LSTM .
    • Converts hardswish subgraphs into atomic ops.
  • Add support for defaulting the value of cycle_length argument of tf.data.Dataset.interleave to the number of schedulable CPU cores.
  • parallel_for : Add converter for MatrixDiag .
  • Add narrow_range attribute to QuantizeAndDequantizeV2 and V3.
  • Added new op: tf.strings.unsorted_segment_join .
  • Add HW acceleration support for topK_v2 .
  • Add new TypeSpec classes.
  • CloudBigtable version updated to v0.10.0.
  • Expose Head as public API.
  • Update docstring for gather to properly describe the non-empty batch_dims case.
  • Added tf.sparse.from_dense utility function.
  • Improved ragged tensor support in TensorFlowTestCase .
  • Makes the a-normal form transformation in Pyct configurable as to which nodes are converted to variables and which are not.
  • ResizeInputTensor now works for all delegates.
  • Add EXPAND_DIMS support to NN API delegate TEST: expand_dims_test
  • tf.cond emits a StatelessIf op if the branch functions are stateless and do not touch any resources.
  • tf.cond , tf.while and if and while in AutoGraph now accept a nonscalar predicate if has a single element. This does not affect non-V2 control flow.
  • tf.while_loop emits a StatelessWhile op if the cond and body functions are stateless and do not touch any resources.
  • Refactors code in Quant8 LSTM support to reduce TFLite binary size.
  • Add support of local soft device placement for eager op.
  • Add HW acceleration support for LogSoftMax .
  • Added a function nested_value_rowids for ragged tensors.
  • Add guard to avo >tf.math.cumulative_logsumexp operation .
  • Add tf.ragged.stack .
  • Fix memory allocation problem when calling AddNewInputConstantTensor .
  • Delegate application failure leaves interpreter in valid state.
  • Add check for correct memory alignment to MemoryAllocation::MemoryAllocation() .
  • Extracts NNAPIDelegateKernel from nnapi_delegate.cc
  • Added support for FusedBatchNormV3 in converter.
  • A ragged to dense op for directly calculating tensors.
  • Fix accidental quadratic graph construction cost in graph-mode tf.gradients() .
  • The precision_mode argument to TrtGraphConverter is now case insensitive.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

a6802739, Aaron Ma, Abdullah Selek, Abolfazl Shahbazi, Ag Ramesh, Albert Z. Guo, Albin Joy, Alex Itkes, Alex Sergeev, Alexander Pivovarov, Alexey Romanov, alhkad, Amit Srivastava, amoitra, Andrew Lihonosov, Andrii Prymostka, Anuj Rawat, Astropeak, Ayush Agrawal, Bairen Yi, Bas Aarts, Bastian Eichenberger, Ben Barsdell, Benjamin Peterson, bhack, Bharat Raghunathan, Bhavani Subramanian, Bryan Cutler, candy.dc, Cao Zongyan, Captain-Pool, Casper Da Costa-Luis, Chen Guoyin, Cheng Chang, chengchingwen, Chong Yan, Choong Yin Thong, Christopher Yeh, Clayne Robison, Coady, Patrick, Dan Ganea, David Norman, Denis Khalikov, Deven Desai, Diego Caballero, Duncan Dean, Duncan Riach, Dwight J Lyle, Eamon Ito-Fisher, eashtian3, EFanZh, ejot, Elroy Ashtian Jr, Eric Schweitz, Fangjun Kuang, Fei Hu, fo40225, formath, Fred Reiss, Frederic Bastien, Fredrik Knutsson, G. Hussain Chinoy, Gabriel, gehring, George Grzegorz Pawelczak, Gianluca Varisco, Gleb Popov, Greg Peatfield, Guillaume Klein, Gurpreet Singh, Gustavo Lima Chaves, haison, Haraldur TóMas HallgríMsson, HarikrishnanBalagopal, HåKon Sandsmark, I-Hong, Ilham Firdausi Putra, Imran Salam, Jason Zaman, Jason Zavaglia, jayhpark530, jefby, Jeff Daily, Jeffrey Poznanovic, Jekyll Lai, Jeroen BéDorf, Jerry Shih, jerryyin, jiakai, JiangXIAO, Joe Bowser, Joel Shapiro, Johan Gunnarsson, Jojimon Varghese, Joon, Josh Beal, Julian Niedermeier, Jun Wan, Junqin Zhang, Junyuan Xie, Justin Tunis, Kaixi Hou, Karl Lessard, Karthik Muthuraman, Kbhute-Ibm, khanhlvg, Koock Yoon, kstuedem, Kyuwon Kim, Lakshay Tokas, leike666666, leonard951, Leslie-Fang, Leslie-Fang-Intel, Li, Guizi, Lukas Folle, Lukas Geiger, Mahmoud Abuzaina, Manraj Singh Grover, Margaret Maynard-Reid, Mark Ryan, Matt Conley, Matthew Bentham, Matthew Denton, mbhuiyan, mdfaijul, Mei Jie, merturl, MichaelKonobeev, Michal W. Tarnowski, minds, mpppk, musikisomorphie, Nagy Mostafa, Nayana Thorat, Neil, Niels Ole Salscheider, Niklas SilfverströM, Niranjan Hasabnis, ocjosen, olramde, Pariksheet Pinjari, Patrick J. Lopresti, Patrik Gustavsson, per1234, PeterLee, Phan Van Nguyen Duc, Phillip Kravtsov, Pooya Davoodi, Pranav Marathe, Putra Manggala, Qingqing Cao, Rajeshwar Reddy T, Ramon ViñAs, Rasmus Diederichsen, Reuben Morais, richardbrks, robert, RonLek, Ryan Jiang, saishruthi, Saket Khandelwal, Saleem Abdulrasool, Sami Kama, Sana-Damani, Sergii Khomenko, Severen Redwood, Shubham Goyal, Sigrid Keydana, Siju Samuel, sleighsoft, smilu97, Son Tran, Srini511, srinivasan.narayanamoorthy, Sumesh Udayakumaran, Sungmann Cho, Tae-Hwan Jung, Taehoon Lee, Takeshi Watanabe, TengLu, terryky, TheMindVirus, ThisIsIsaac, Till Hoffmann, Timothy Liu, Tomer Gafner, Tongxuan Liu, Trent Lo, Trevor Morris, Uday Bondhugula, Vasileios Lioutas, vbvg2008, Vishnuvardhan Janapati, Vivek Suryamurthy, Wei Wang, Wen-Heng (Jack) Chung, wenxizhu, William D. Irons, winstonq, wyzhao, Xiaoming (Jason) Cui, Xinan Jiang, Xinping Wang, Yann-Yy, Yasir Modak, Yong Tang, Yongfeng Gu, Yuchen Ying, Yuxin Wu, zyeric, 王振华 (Zhenhua Wang)

goldiegadde released this Oct 1, 2020 · 32 commits to r1.15 since this release

Release 1.15.0-rc2

This is the last 1.x release for TensorFlow. We do not expect to update the 1.x branch with features, although we will issue patch releases to fix vulnerabilities for at least one year.

Major Features and Improvements

  • As announced, tensorflow pip package will by default include GPU support (same as tensorflow-gpu now) for the platforms we currently have GPU support (Linux and Windows). It will work on machines with and without Nvidia GPUs. tensorflow-gpu will still be available, and CPU-only packages can be downloaded at tensorflow-cpu for users who are concerned about package size.
  • TensorFlow 1.15 contains a complete implementation of the 2.0 API in its compat.v2 module . It contains a copy of the 1.15 main module (without contrib ) in the compat.v1 module . TensorFlow 1.15 is able to emulate 2.0 behavior using the enable_v2_behavior() function.
    This enables writing forward compatible code: by explicitly importing either tensorflow.compat.v1 or tensorflow.compat.v2 , you can ensure that your code works without modifications against an installation of 1.15 or 2.0.
  • EagerTensor now supports numpy buffer interface for tensors.
  • Add toggles tf.enable_control_flow_v2() and tf.disable_control_flow_v2() for enabling/disabling v2 control flow.
  • Enable v2 control flow as part of tf.enable_v2_behavior() and TF2_BEHAVIOR=1 .
  • AutoGraph translates Python control flow into TensorFlow expressions, allowing users to write regular Python inside tf.function -decorated functions. AutoGraph is also applied in functions used with tf.data , tf.distribute and tf.keras APIS.
  • Adds enable_tensor_equality() , which switches the behavior such that:
    • Tensors are no longer hashable.
    • Tensors can be compared with == and !=, yielding a Boolean Tensor with element-wise comparison results. This will be the default behavior in 2.0.
  • Auto Mixed-Precision graph optimizer simplifies converting models to float16 for acceleration on Volta and Turing Tensor Cores. This feature can be enabled by wrapping an optimizer class with tf.train.experimental.enable_mixed_precision_graph_rewrite() .
  • Add environment variable TF_CUDNN_DETERMINISTIC . Setting to «true» or «1» forces the selection of deterministic cuDNN convolution and max-pooling algorithms. When this is enabled, the algorithm selection procedure itself is also deterministic.
  • TensorRT
    • Migrate TensorRT conversion sources from contrib to compiler directory in preparation for TF 2.0.
    • Add additional, user friendly TrtGraphConverter API for TensorRT conversion.
    • Expand support for TensorFlow operators in TensorRT conversion (e.g.
      Gather , Slice , Pack , Unpack , ArgMin , ArgMax , DepthSpaceShuffle ).
    • Support TensorFlow operator CombinedNonMaxSuppression in TensorRT conversion which
      significantly accelerates object detection models.


Breaking Changes

  • Tensorflow code now produces 2 different pip packages: tensorflow_core containing all the code (in the future it will contain only the private implementation) and tensorflow which is a virtual pip package doing forwarding to tensorflow_core (and in the future will contain only the public API of tensorflow). We don’t expect this to be breaking, unless you were importing directly from the implementation.
  • TensorFlow 1.15 is built using devtoolset7 (GCC7) on Ubuntu 16. This may lead to ABI incompatibilities with extensions built against earlier versions of TensorFlow.
  • Deprecated the use of constraint= and .constraint with ResourceVariable.
  • tf.keras :
    • OMP_NUM_THREADS is no longer used by the default Keras config. To configure the number of threads, use tf.config.threading APIs.
    • tf.keras.model.save_model and model.save now defaults to saving a TensorFlow SavedModel.
    • keras.backend.resize_images (and consequently, keras.layers.Upsampling2D ) behavior has changed, a bug in the resizing implementation was fixed.
    • Layers now default to float32 , and automatically cast their inputs to the layer’s dtype. If you had a model that used float64 , it will probably silently use float32 in TensorFlow2, and a warning will be issued that starts with Layer «layer-name» is casting an input tensor from dtype float64 to the layer’s dtype of float32. To fix, either set the default dtype to float64 with tf.keras.backend.set_floatx(‘float64′) , or pass dtype=’float64’ to each of the Layer constructors. See tf.keras.layers.Layer for more information.
    • Some tf.assert_* methods now raise assertions at operation creation time (i.e. when this Python line executes) if the input tensors’ values are known at that time, not during the session.run(). When this happens, a noop is returned and the input tensors are marked non-feedable. In other words, if they are used as keys in feed_dict argument to session.run() , an error will be raised. Also, because some assert ops don’t make it into the graph, the graph structure changes. A different graph can result in different per-op random seeds when they are not given explicitly (most often).

Bug Fixes and Other Changes

  • tf.estimator :
    • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format, which allows the saved checkpoints to be compatible with model.load_weights .
    • Fix tests in canned estimators.
    • Expose Head as public API.
    • Fixes critical bugs that help with DenseFeatures usability in TF2
  • tf.data :
    • Promoting unbatch from experimental to core API.
    • Adding support for datasets as inputs to from_tensors and from_tensor_slices and batching and unbatching of nested datasets.
  • tf.keras :
    • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format, which allows the saved checkpoints to be compatible with model.load_weights .
    • Saving a Keras Model using tf.saved_model.save now saves the list of variables, trainable variables, regularization losses, and the call function.
    • Deprecated tf.keras.experimental.export_saved_model and tf.keras.experimental.function . Please use tf.keras.models.save_model(. save_format=’tf’) and tf.keras.models.load_model instead.
    • Add an implementation=3 mode for tf.keras.layers.LocallyConnected2D and tf.keras.layers.LocallyConnected1D layers using tf.SparseTensor to store weights, allowing a dramatic speedup for large sparse models.
    • Enable the Keras compile API experimental_run_tf_function flag by default. This flag enables single training/eval/predict execution path. With this 1. All input types are converted to Dataset . 2. When distribution strategy is not specified this goes through the no-op distribution strategy path. 3. Execution is wrapped in tf.function unless run_eagerly=True is set in compile.
    • Raise error if batch_size argument is used when input is dataset/generator/keras sequence.
  • tf.lite
    • Add GATHER support to NN API delegate.
    • tflite object detection script has a debug mode.
    • Add delegate support for QUANTIZE .
    • Added evaluation script for COCO minival.
    • Add delegate support for QUANTIZED_16BIT_LSTM .
    • Converts hardswish subgraphs into atomic ops.
  • Add support for defaulting the value of cycle_length argument of tf.data.Dataset.interleave to the number of schedulable CPU cores.
  • parallel_for : Add converter for MatrixDiag .
  • Add narrow_range attribute to QuantizeAndDequantizeV2 and V3.
  • Added new op: tf.strings.unsorted_segment_join .
  • Add HW acceleration support for topK_v2 .
  • Add new TypeSpec classes.
  • CloudBigtable version updated to v0.10.0.
  • Expose Head as public API.
  • Update docstring for gather to properly describe the non-empty batch_dims case.
  • Added tf.sparse.from_dense utility function.
  • Improved ragged tensor support in TensorFlowTestCase .
  • Makes the a-normal form transformation in Pyct configurable as to which nodes are converted to variables and which are not.
  • ResizeInputTensor now works for all delegates.
  • Add EXPAND_DIMS support to NN API delegate TEST: expand_dims_test
  • tf.cond emits a StatelessIf op if the branch functions are stateless and do not touch any resources.
  • tf.cond , tf.while and if and while in AutoGraph now accept a nonscalar predicate if has a single element. This does not affect non-V2 control flow.
  • tf.while_loop emits a StatelessWhile op if the cond and body functions are stateless and do not touch any resources.
  • Refactors code in Quant8 LSTM support to reduce TFLite binary size.
  • Add support of local soft device placement for eager op.
  • Add HW acceleration support for LogSoftMax .
  • Added a function nested_value_rowids for ragged tensors.
  • Add guard to avo >tf.math.cumulative_logsumexp operation .
  • Add tf.ragged.stack .
  • Fix memory allocation problem when calling AddNewInputConstantTensor .
  • Delegate application failure leaves interpreter in valid state.
  • Add check for correct memory alignment to MemoryAllocation::MemoryAllocation() .
  • Extracts NNAPIDelegateKernel from nnapi_delegate.cc
  • Added support for FusedBatchNormV3 in converter.
  • A ragged to dense op for directly calculating tensors.
  • Fix accidental quadratic graph construction cost in graph-mode tf.gradients() .
  • The precision_mode argument to TrtGraphConverter is now case insensitive.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

a6802739, Aaron Ma, Abdullah Selek, Abolfazl Shahbazi, Ag Ramesh, Albert Z. Guo, Albin Joy, Alex Itkes, Alex Sergeev, Alexander Pivovarov, Alexey Romanov, alhkad, Amit Srivastava, amoitra, Andrew Lihonosov, Andrii Prymostka, Anuj Rawat, Astropeak, Ayush Agrawal, Bairen Yi, Bas Aarts, Bastian Eichenberger, Ben Barsdell, Benjamin Peterson, bhack, Bharat Raghunathan, Bhavani Subramanian, Bryan Cutler, candy.dc, Cao Zongyan, Captain-Pool, Casper Da Costa-Luis, Chen Guoyin, Cheng Chang, chengchingwen, Chong Yan, Choong Yin Thong, Christopher Yeh, Clayne Robison, Coady, Patrick, Dan Ganea, David Norman, Denis Khalikov, Deven Desai, Diego Caballero, Duncan Dean, Duncan Riach, Dwight J Lyle, Eamon Ito-Fisher, eashtian3, EFanZh, ejot, Elroy Ashtian Jr, Eric Schweitz, Fangjun Kuang, Fei Hu, fo40225, formath, Fred Reiss, Frederic Bastien, Fredrik Knutsson, G. Hussain Chinoy, Gabriel, gehring, George Grzegorz Pawelczak, Gianluca Varisco, Gleb Popov, Greg Peatfield, Guillaume Klein, Gurpreet Singh, Gustavo Lima Chaves, haison, Haraldur TóMas HallgríMsson, HarikrishnanBalagopal, HåKon Sandsmark, I-Hong, Ilham Firdausi Putra, Imran Salam, Jason Zaman, Jason Zavaglia, jayhpark530, jefby, Jeff Daily, Jeffrey Poznanovic, Jekyll Lai, Jeroen BéDorf, Jerry Shih, jerryyin, jiakai, JiangXIAO, Joe Bowser, Joel Shapiro, Johan Gunnarsson, Jojimon Varghese, Joon, Josh Beal, Julian Niedermeier, Jun Wan, Junqin Zhang, Junyuan Xie, Justin Tunis, Kaixi Hou, Karl Lessard, Karthik Muthuraman, Kbhute-Ibm, khanhlvg, Koock Yoon, kstuedem, Kyuwon Kim, Lakshay Tokas, leike666666, leonard951, Leslie-Fang, Leslie-Fang-Intel, Li, Guizi, Lukas Folle, Lukas Geiger, Mahmoud Abuzaina, Manraj Singh Grover, Margaret Maynard-Reid, Mark Ryan, Matt Conley, Matthew Bentham, Matthew Denton, mbhuiyan, mdfaijul, Mei Jie, merturl, MichaelKonobeev, Michal W. Tarnowski, minds, mpppk, musikisomorphie, Nagy Mostafa, Nayana Thorat, Neil, Niels Ole Salscheider, Niklas SilfverströM, Niranjan Hasabnis, ocjosen, olramde, Pariksheet Pinjari, Patrick J. Lopresti, Patrik Gustavsson, per1234, PeterLee, Phan Van Nguyen Duc, Phillip Kravtsov, Pooya Davoodi, Pranav Marathe, Putra Manggala, Qingqing Cao, Rajeshwar Reddy T, Ramon ViñAs, Rasmus Diederichsen, Reuben Morais, richardbrks, robert, RonLek, Ryan Jiang, saishruthi, Saket Khandelwal, Saleem Abdulrasool, Sami Kama, Sana-Damani, Sergii Khomenko, Severen Redwood, Shubham Goyal, Sigrid Keydana, Siju Samuel, sleighsoft, smilu97, Son Tran, Srini511, srinivasan.narayanamoorthy, Sumesh Udayakumaran, Sungmann Cho, Tae-Hwan Jung, Taehoon Lee, Takeshi Watanabe, TengLu, terryky, TheMindVirus, ThisIsIsaac, Till Hoffmann, Timothy Liu, Tomer Gafner, Tongxuan Liu, Trent Lo, Trevor Morris, Uday Bondhugula, Vasileios Lioutas, vbvg2008, Vishnuvardhan Janapati, Vivek Suryamurthy, Wei Wang, Wen-Heng (Jack) Chung, wenxizhu, William D. Irons, winstonq, wyzhao, Xiaoming (Jason) Cui, Xinan Jiang, Xinping Wang, Yann-Yy, Yasir Modak, Yong Tang, Yongfeng Gu, Yuchen Ying, Yuxin Wu, zyeric, 王振华 (Zhenhua Wang)

goldiegadde released this Sep 30, 2020 · 9 commits to r2.0 since this release

Release 2.0.0

Major Features and Improvements

TensorFlow 2.0 focuses on simplicity and ease of use, featuring updates like:

  • Easy model building with Keras and eager execution.
  • Robust model deployment in production on any platform.
  • Powerful experimentation for research.
  • API simplification by reducing duplication and removing deprecated endpoints.

For details on best practices with 2.0, see the Effective 2.0 guide

For information on upgrading your existing TensorFlow 1.x models, please refer to our Upgrade and Migration guides. We have also released a collection of tutorials and getting started guides.

Highlights

  • TF 2.0 delivers Keras as the central high level API used to build and train models. Keras provides several model-building APIs such as Sequential, Functional, and Subclassing along with eager execution, for immediate iteration and intuitive debugging, and tf.data , for building scalable input pipelines. Checkout guide for additional details.
  • Distribution Strategy: TF 2.0 users will be able to use the tf.distribute.Strategy API to distribute training with minimal code changes, yielding great out-of-the-box performance. It supports distributed training with Keras model.fit, as well as with custom training loops. Multi-GPU support is available, along with experimental support for multi worker and Cloud TPUs. Check out the guide for more details.
  • Functions, not Sessions. The traditional declarative programming model of building a graph and executing it via a tf.Session is discouraged, and replaced with by writing regular Python functions. Using the tf.function decorator, such functions can be turned into graphs which can be executed remotely, serialized, and optimized for performance.
  • Unification of tf.train.Optimizers and tf.keras.Optimizers . Use tf.keras.Optimizers for TF2.0. compute_gradients is removed as public API, use GradientTape to compute gradients.
  • AutoGraph translates Python control flow into TensorFlow expressions, allowing users to write regular Python inside tf.function -decorated functions. AutoGraph is also applied in functions used with tf.data, tf.distribute and tf.keras APIs.
  • Unification of exchange formats to SavedModel. All TensorFlow ecosystem projects (TensorFlow Lite, TensorFlow JS, TensorFlow Serving, TensorFlow Hub) accept SavedModels. Model state should be saved to and restored from SavedModels.
  • API Changes: Many API symbols have been renamed or removed, and argument names have changed. Many of these changes are motivated by consistency and clarity. The 1.x API remains available in the compat.v1 module. A list of all symbol changes can be found here.
  • API clean-up, included removing tf.app , tf.flags , and tf.logging in favor of absl-py.
  • No more global variables with helper methods like tf.global_variables_initializer and tf.get_global_step .
  • Add toggles tf.enable_control_flow_v2() and tf.disable_control_flow_v2() for enabling/disabling v2 control flow.
  • Enable v2 control flow as part of tf.enable_v2_behavior() and TF2_BEHAVIOR=1 .
  • Fixes autocomplete for most TensorFlow API references by switching to use relative imports in API __init__.py files.
  • Auto Mixed-Precision graph optimizer simplifies converting models to float16 for acceleration on Volta and Turing Tensor Cores. This feature can be enabled by wrapping an optimizer class with tf.train.experimental.enable_mixed_precision_graph_rewrite() .

  • Add environment variable TF_CUDNN_DETERMINISTIC . Setting to TRUE or «1» forces the selection of deterministic cuDNN convolution and max-pooling algorithms. When this is enabled, the algorithm selection procedure itself is also deterministic.

Breaking Changes

Many backwards incompatible API changes have been made to clean up the APIs and make them more consistent.

  • TensorFlow 2.0.0 is built using devtoolset7 (GCC7) on Ubuntu 16. This may lead to ABI incompatibilities with extensions built against earlier versions of TensorFlow.
  • Tensorflow code now produces 2 different pip packages: tensorflow_core containing all the code (in the future it will contain only the private implementation) and tensorflow which is a virtual pip package doing forwarding to tensorflow_core (and in the future will contain only the public API of tensorflow). We don’t expect this to be breaking, unless you were importing directly from the implementation.
    Removed the freeze_graph command line tool; SavedModel should be used in place of frozen graphs.
  • tf.contrib has been deprecated, and functionality has been either migrated to the core TensorFlow API, to an ecosystem project such as tensorflow/addons or tensorflow/io, or removed entirely.
  • Remove tf.contrib.timeseries dependency on TF distributions.
  • Replace contrib references with tf.estimator.experimental.* for apis in early_stopping.py .
  • Premade estimators in the tf.estimator.DNN/Linear/DNNLinearCombined family have been updated to use tf.keras.optimizers instead of the tf.compat.v1.train.Optimizer s. If you do not pass in an optimizer= arg or if you use a string, the premade estimator will use the Keras optimizer. This is checkpoint breaking, as the optimizers have separate variables. A checkpoint converter tool for converting optimizers is included with the release, but if you want to avoid any change, switch to the v1 version of the estimator: tf.compat.v1.estimator.DNN/Linear/DNNLinearCombined* .
  • Default aggregation for canned Estimators is now SUM_OVER_BATCH_SIZE . To maintain previous default behavior, please pass SUM as the loss aggregation method.
  • Canned Estimators don’t support input_layer_partitioner arg in the API. If you have this arg, you will have to switch to tf.compat.v1 canned Estimators .
  • Estimator.export_savedmodel has been renamed to export_saved_model .
  • When saving to SavedModel, Estimators will strip default op attributes. This is almost always the correct behavior, as it is more forwards compatible, but if you require that default attributes to be saved with the model, please use tf.compat.v1.Estimator .
  • Feature Columns have been upgraded to be more Eager-friendly and to work with Keras. As a result, tf.feature_column.input_layer has been deprecated in favor of tf.keras.layers.DenseFeatures . v1 feature columns have direct analogues in v2 except for shared_embedding_columns , which are not cross-compatible with v1 and v2. Use tf.feature_column.shared_embeddings instead.
  • OMP_NUM_THREADS is no longer used by the default Keras config. To configure the number of threads, use tf.config.threading APIs.
  • tf.keras.model.save_model and model.save now defaults to saving a TensorFlow SavedModel. HDF5 files are still supported.
  • Deprecated tf.keras.experimental.export_saved_model and tf.keras.experimental.function . Please use tf.keras.models.save_model(. save_format=’tf’) and tf.keras.models.load_model instead.
  • Layers now default to float32, and automatically cast their inputs to the layer’s dtype. If you had a model that used float64, it will probably silently use float32 in TensorFlow 2, and a warning will be issued that starts with Layer is casting an input tensor from dtype float64 to the layer’s dtype of float32. To fix, either set the default dtype to float64 with tf.keras.backend.set_floatx(‘float64′) , or pass dtype=’float64’ to each of the Layer constructors. See tf.keras.layers.Layer for more information.
  • Removed lite.OpHint , lite.experimental , and lite.constant from 2.0 API.

Tensors are no longer hashable, but instead compare element-wise with == and != . Use tf.compat.v1.disable_tensor_equality() to return to the previous behavior.

Performing equality operations on Tensors or Variables with incompatible shapes an exception is no longer thrown. Instead __eq__ returns False and __ne__ returns True.

Removed tf.string_split from v2 API.

Deprecated the use of constraint= and .constraint with ResourceVariable.

Add UnifiedGRU as the new GRU implementation for tf2.0. Change the default recurrent activation function for GRU from hard_sigmoid to sigmoid , and reset_after to True in 2.0. Historically recurrent activation is hard_sigmoid since it is fast than ‘sigmo , reset_after=False) to fallback to 1.x behavior.

CUDNN_INSTALL_PATH , TENSORRT_INSTALL_PATH , NCCL_INSTALL_PATH , NCCL_HDR_PATH are deprecated. Use TF_CUDA_PATHS instead which supports a comma-separated list of base paths that are searched to find CUDA libraries and headers.

Refer to our public project status tracker and issues tagged with 2.0 on GitHub for insight into recent issues and development progress.

If you experience any snags when using TF 2.0, please let us know at the TF 2.0 Testing User Group. We have a support mailing list as well as weekly testing meetings, and would love to hear your migration feedback and questions.

Bug Fixes and Other Changes

  • Expose tf.contrib.proto.* ops in tf.io (they will exist in TF2)
  • Add support for TensorArrays to tf.data Dataset .
  • Integrate Ragged Tensors with tf.data .
  • All core and experimental tf.data transformations that input user-defined functions can span multiple devices now.
  • Extending the TF 2.0 support for shuffle(. reshuffle_each_iteration=True) and cache() to work across different Python iterators for the same dataset.
  • Removing the experimental_numa_aware option from tf.data.Options .
  • Add num_parallel_reads and passing in a Dataset containing filenames into TextLineDataset and FixedLengthRecordDataset .
  • Add support for defaulting the value of cycle_length argument of tf.data.Dataset.interleave to the number of schedulable CPU cores.
  • Promoting tf.data.experimental.enumerate_dataset to core as tf.data.Dataset.enumerate .
  • Promoting tf.data.experimental.unbatch to core as tf.data.Dataset.unbatch .
  • Adds option for introducing slack in the pipeline to reduce CPU contention, via tf.data.Options().experimental_slack = True
  • Added experimental support for parallel batching to batch() and padded_batch() . This functionality can be enabled through tf.data.Options() .
  • Support cancellation of long-running reduce .
  • Now we use dataset node name as prefix instead of the op name, to identify the component correctly in metrics, for pipelines with repeated components.
  • Improve the performance of datasets using from_tensors() .
  • Promoting unbatch from experimental to core API.
  • Adding support for datasets as inputs to from_tensors and from_tensor_slices and batching and unbatching of nested datasets.
  • Enable tf.distribute.experimental.MultiWorkerMirroredStrategy working in eager mode.
  • Callbacks are supported in MultiWorkerMirroredStrategy .
  • Disable run_eagerly and distribution strategy if there are symbolic tensors added to the model using add_metric or add_loss .
  • Loss and gradients should now more reliably be correctly scaled w.r.t. the global batch size when using a tf.distribute.Strategy .
  • Set default loss reduction as AUTO for improving reliability of loss scaling with distribution strategy and custom training loops. AUTO indicates that the reduction option will be determined by the usage context. For almost all cases this defaults to SUM_OVER_BATCH_SIZE . When used in distribution strategy scope, outside of built-in training loops such as tf.keras compile and fit , we expect reduction value to be ‘None’ or ‘SUM’. Using other values will raise an error.
  • Support for multi-host ncclAllReduce in Distribution Strategy.
  • Replace tf.contrib.estimator.add_metrics with tf.estimator.add_metrics
  • Use tf.compat.v1.estimator.inputs instead of tf.estimator.inputs
  • Replace contrib references with tf.estimator.experimental.* for apis in early_s in Estimator
  • Canned Estimators will now use keras optimizers by default. An error will be raised if tf.train.Optimizers are used, and you will have to switch to tf.keras.optimizers or tf.compat.v1 canned Estimators.
  • A checkpoint converter for canned Estimators has been provided to transition canned Estimators that are warm started from tf.train.Optimizers to tf.keras.optimizers .
  • Losses are scaled in canned estimator v2 and not in the optimizers anymore. If you are using Estimator + distribution strategy + optimikzer v1 then the behavior does not change. This implies that if you are using custom estimator with optimizer v2, you have to scale losses. We have new utilities to help scale losses tf.nn.compute_average_loss , tf.nn.scale_regularization_loss .
  • Premade models (including Linear and WideDeep) have been introduced for the purpose of replacing Premade estimators.
  • Model saving changes
  • model.save and tf.saved_model.save may now save to the TensorFlow SavedModel format. The model can be restored using tf.keras.models.load_model . HDF5 files are still supported, and may be used by specifying save_format=»h5″ when saving.
  • Raw TensorFlow functions can now be used in conjunction with the Keras Functional API during model creation. This obviates the need for users to create Lambda layers in most cases when using the Functional API. Like Lambda layers, TensorFlow functions that result in Variable creation or assign ops are not supported.
  • Add support for passing list of lists to the metrics argument in Keras compile .
  • Add tf.keras.layers.AbstractRNNCell as the preferred implementation for RNN cells in TF v2. User can use it to implement RNN cells with custom behavior.
  • Keras training and validation curves are shown on the same plot when using the TensorBoard callback.
  • Switched Keras fit/evaluate/predict execution to use only a single unified path by default unless eager execution has been explicitly disabled, regardless of input type. This unified path places an eager-friendly training step inside of a tf.function . With this
  1. All input types are converted to Dataset .
  2. The path assumes there is always a distribution strategy. when distribution strategy is not specified the path uses a no-op distribution strategy.
  3. The training step is wrapped in tf.function unless run_eagerly=True is set in compile. The single path execution code does not yet support all use cases. We fallback to the existing v1 execution paths if your model contains the following:
  4. sample_weight_mode in compile
  5. weighted_metrics in compile
  6. v1 optimizer
  7. target tensors in compile
    If you are experiencing any issues because of this change, please inform us (file an issue) about your use case and you can unblock yourself by setting experimental_run_tf_function=False in compile meanwhile. We have seen couple of use cases where the model usage pattern is not as expected and would not work with this change.
  8. output tensors of one layer is used in the constructor of another.
  9. symbolic tensors outside the scope of the model are used in custom loss functions.
    The flag can be disabled for these cases and ideally the usage pattern will need to be fixed.
  • Mark Keras set_session as compat.v1 only.
  • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format , which allows the saved checkpoints to be compatible with model.load_weights .
  • keras.backend.resize_images (and consequently, keras.layers.Upsampling2D ) behavior has changed, a bug in the resizing implementation was fixed.
  • Add an implementation=3 mode for tf.keras.layers.LocallyConnected2D and tf.keras.layers.LocallyConnected1D layers using tf.SparseTensor to store weights, allowing a dramatic speedup for large sparse models.
  • Raise error if batch_size argument is used when input is dataset/generator/keras sequence.
  • Update TF 2.0 keras.backend.name_scope to use TF 2.0 name_scope .
  • Add v2 module aliases for losses, metrics, initializers and optimizers: tf.losses = tf.keras.losses & tf.metrics = tf.keras.metrics & tf.initializers = tf.keras.initializers & tf.optimizers = tf.keras.optimizers .
  • Updates binary cross entropy logic in Keras when input is probabilities. Instead of converting probabilities to logits, we are using the cross entropy formula for probabilities.
  • Added public APIs for cumsum and cumprod keras backend functions.
  • Add support for temporal sample weight mode in subclassed models.
  • Raise ValueError if an integer is passed to the training APIs.
  • Added fault-tolerance support for training Keras model via model.fit() with MultiWorkerMirroredStrategy , tutorial available.
  • Custom Callback tutorial is now available.
  • To train with tf.distribute , Keras API is recommended over estimator.
  • steps_per_epoch and steps arguments are supported with numpy arrays.
  • New error message when unexpected keys are used in sample_weight/class_weight dictionaries
  • Losses are scaled in Keras compile/fit and not in the optimizers anymore. If you are using custom training loop, we have new utilities to help scale losses tf.nn.compute_average_loss , tf.nn.scale_regularization_loss .
  • Layer apply and add_variable APIs are deprecated.
  • Added support for channels first data format in cross entropy losses with logits and support for tensors with unknown ranks.
  • Error messages will be raised if add_update , add_metric , add_loss , activity regularizers are used inside of a control flow branch.
  • New loss reduction types:
    1. AUTO : Indicates that the reduction option will be determined by the usage context. For almost all cases this defaults to SUM_OVER_BATCH_SIZE . When used with tf.distribute.Strategy , outside of built-in training loops such as tf.keras compile and fit , we expect reduction value to be SUM or NONE . Using AUTO in that case will raise an error.
    2. NONE : Weighted losses with one dimension reduced (axis=-1, or axis specified by loss function). When this reduction type used with built-in Keras training loops like fit / evaluate , the unreduced vector loss is passed to the optimizer but the reported loss will be a scalar value.
    3. SUM : Scalar sum of weighted losses. 4. SUM_OVER_BATCH_SIZE : Scalar SUM divided by number of elements in losses. This reduction type is not supported when used with tf.distribute.Strategy outside of built-in training loops like tf.keras compile / fit .
  • Wraps losses passed to the compile API (strings and v1 losses) which are not instances of v2 Loss class in LossWrapper class. => All losses will now use SUM_OVER_BATCH_SIZE reduction as default.
  • model.add_loss(symbolic_tensor) should work in ambient eager.
  • Update metric name to always reflect what the user has given in compile. Affects following cases
    1. When name is given as ‘accuracy’/’crossentropy’
    2. When an aliased function name is used eg. ‘mse’
    3. Removing the weighted prefix from weighted metric names.
  • Allow non-Tensors through v2 losses.
  • Add v2 sparse categorical crossentropy metric.
  • Add v2 APIs for AUCCurve and AUCSummationMethod enums.
  • add_update can now be passed a zero-arg callable in order to support turning off the update when setting trainable=False on a Layer of a Model compiled with run_eagerly=True .
  • Standardize the LayerNormalization API by replacing the args norm_axis and params_axis with axis .
  • Fixed critical bugs that help with DenseFeatures usability in TF2

  • Added evaluation script for COCO minival
  • Add delegate support for QUANTIZE .
  • Add GATHER support to NN API delegate.
  • Added support for TFLiteConverter Python API in 2.0. Contains functions from_saved_model, from_keras_file, and from_concrete_functions.
  • Add EXPAND_DIMS support to NN API delegate TEST.
  • Add narrow_range attribute to QuantizeAndDequantizeV2 and V3.
  • Added support for tflite_convert command line tool in 2.0.
  • Post-training quantization tool supports quantizing weights shared by multiple operations. The models made with versions of this tool will use INT8 types for weights and will only be executable interpreters from this version onwards.
  • Post-training quantization tool supports fp16 weights and GPU delegate acceleration for fp16.
  • Add delegate support for QUANTIZED_16BIT_LSTM .
  • Extracts NNAPIDelegateKernel from nnapi_delegate.cc
  • Add TensorFlow 2.0-compatible TrtGraphConverterV2 API for TensorRT conversion.
    TensorRT initialization arguments are now passed wrapped in a named-tuple,
    TrtConversionParams , rather than as separate arguments as in TrtGraphConverter .
  • Changed API to optimize TensorRT enginges during graph optimization. This is now
    done by calling converter.build() where previously is_dynamic_op=False would
    be set.
  • converter.convert() no longer returns a tf.function . Now the funtion must be
    accessed from the saved model.
  • The converter.calibrate() method has been removed. To trigger calibration, a
    calibration_input_fn should be provided to converter.convert() .
  • Fix accidental quadratic graph construction cost in graph-mode tf.gradients() .
  • ResourceVariable’s gather op supports batch dimensions.
  • ResourceVariable support for gather_nd .
  • ResourceVariable and Variable no longer accepts constraint in the constructor, nor expose it as a @Property.
  • Added gradient for SparseToDense op.
  • Expose a flag that allows the number of threads to vary across Python benchmarks.
  • image.resize in 2.0 now supports gradients for the new resize kernels.
  • image.resize now considers proper pixel centers and has new kernels (incl. anti-aliasing).
  • Renamed tf.image functions to remove duplicate «image» where it is redundant.
  • Variadic reduce is supported on CPU Variadic reduce is supported on CPU
  • Remove unused StringViewVariantWrapper .
  • Delete unused Fingerprint64Map op registration
  • Add broadcasting support to tf.matmul .
  • Add C++ Gradient for BatchMatMulV2 .
  • Add tf.math.cumulative_logsumexp operation.
  • Add ellipsis (. ) support for tf.einsum() .
  • Add expand_composites argument to all nest.* methods.
  • Added strings.byte_split .
  • Add a new «result_type» parameter to tf.strings.split .
  • Add name argument to tf.string_split and tf.strings_split .
  • Extend tf.strings.split to support inputs with any rank.
  • Added tf.random.binomial .
  • Added key and skip methods to random.experimental.Generator .
  • Extend tf.function with basic support for CompositeTensors arguments (such as SparseTensor and RaggedTensor ).
  • parallel_for.pfor : add converters for Softmax, LogSoftmax, IsNaN, All, Any, and MatrixSetDiag.
  • parallel_for : add converters for LowerTriangularSolve and Cholesky.
  • parallel_for : add converters for LogMatrixDeterminant and MatrixBandPart .
  • parallel_for : Add converter for MatrixDiag .
  • parallel_for : Add converters for OneHot , LowerBound , UpperBound .
  • parallel_for : add converter for BroadcastTo .
  • Add pfor converter for Squeeze .
  • Add RaggedTensor.placeholder() .
  • Add ragged tensor support to tf.squeeze .
  • Update RaggedTensors to support int32 row_splits.
  • Allow LinearOperator.solve to take a LinearOperator .
  • Allow all dtypes for LinearOperatorCirculant .
  • Introduce MaxParallelism method
  • Add LinearOperatorHouseholder .
  • Adds Philox support to new stateful RNG’s XLA path.
  • Added TensorSpec support for CompositeTensors.
  • Added tf.linalg.tridiagonal_solve op.
  • Added partial_pivoting input parameter to tf.linalg.tridiagonal_solve .
  • Added gradient to tf.linalg.tridiagonal_solve .
  • Added tf.linalg.tridiagonal_mul op .
  • Added GPU implementation of tf.linalg.tridiagonal_matmul .
  • Added LinearOperatorToeplitz .
  • Upgraded LIBXSMM to version 1.11.
  • Uniform processing of quantized embeddings by Gather and EmbeddingLookup Ops.
  • Correct a misstatement in the documentation of the sparse softmax cross entropy logit parameter.
  • Add tf.ragged.boolean_mask .
  • tf.switch_case added, which selects a branch_fn based on a branch_index.
  • The C++ kernel of gather op supports batch dimensions.
  • Fixed default value and documentation for trainable arg of tf.Variable.
  • EagerTensor now supports numpy buffer interface for tensors.
  • This change bumps the version number of the FullyConnected Op to 5.
  • Added new op: tf.strings.unsorted_segment_join .
  • Added HW acceleration support for topK_v2 .
  • CloudBigtable version updated to v0.10.0 BEGIN_PUBLIC CloudBigtable version updated to v0.10.0.
  • Expose Head as public API.
  • Added tf.sparse.from_dense utility function.
  • Improved ragged tensor support in TensorFlowTestCase .
  • Added a function nested_value_rowids for ragged tensors.
  • Added tf.ragged.stack .
  • Makes the a-normal form transformation in Pyct configurable as to which nodes are converted to variables and which are not.
  • ResizeInputTensor now works for all delegates.
  • tf.cond emits a StatelessIf op if the branch functions are stateless and do not touch any resources.
  • Add support of local soft device placement for eager op.
  • Pass partial_pivoting to the _TridiagonalSolveGrad .
  • Add HW acceleration support for LogSoftMax .
  • Add guard to avo >AddNewInputConstantTensor .
  • Delegate application failure leaves interpreter in valid state
  • tf.while_loop emits a StatelessWhile op if the cond and body functions are stateless and do not touch any resources.
  • tf.cond , tf.while and if and while in AutoGraph now accept a nonscalar predicate if has a single element. This does not affect non-V2 control flow.
  • Fix potential security vulnerability where decoding variant tensors from proto could result in heap out of bounds memory access.
  • Only create a GCS directory object if the object does not already exist.
  • Introduce dynamic constructor argument in Layer and Model, which should be set to True when using imperative control flow in the call method.
  • Begin adding Go wrapper for C Eager API.
  • XLA HLO graphs can be inspected with interactive_graphviz tool now.
  • Add dataset ops to the graph (or create kernels in Eager execution) during the python Dataset object creation instead doing it during Iterator creation time.
  • Add batch_dims argument to tf.gather .
  • The behavior of tf.gather is now correct when axis=None and batch_dims .
  • Update docstring for gather to properly describe the non-empty batch_dims case.
  • Removing of dtype in the constructor of initializers and partition_info in call.
  • Add tf.math.nextafter op.
  • Turn on MKL-DNN contraction kernels by default. MKL-DNN dynamically dispatches the best kernel implementation based on CPU vector architecture. To disable them, build with —define=tensorflow_mkldnn_contraction_kernel=0 .
  • tf.linspace(start, stop, num) now always uses «stop» as last value (for num > 1)
  • Added top-k to precision and recall to keras metrics.
  • Add a ragged size op and register it to the op dispatcher
  • Transitive dependencies on : pooling_ops were removed. Some users may need to add explicit dependencies on : pooling_ops if they reference the operators from that library.
  • Add CompositeTensor base class.
  • Malformed gif images could result in an access out of bounds in the color palette of the frame. This has been fixed now
  • Add templates and interfaces for creating lookup tables
  • Tensor::UnsafeCopyFromInternal deprecated in favor Tensor::BitcastFrom .
  • In map_vectorization optimization, reduce the degree of parallelism in the vectorized map node.
  • Add variant wrapper for absl::string_view .
  • Add OpKernels for some stateless maps.
  • DType is no longer convertible to an int. Use dtype.as_datatype_enum instead of int(dtype) to get the same result.
  • Support both binary and -1/1 label input in v2 hinge and squared hinge losses.
  • Added LinearOperator.adjoint and LinearOperator.H (alias).
  • Expose CriticalSection in core as tf.CriticalSection .
  • Enhanced graphviz output.
  • Add opkernel templates for common table operations.
  • Fix callbacks do not log values in eager mode when a deferred build model is used.
  • SignatureDef util functions have been deprecated.
  • Update Fingerprint64Map to use aliases
  • Add legacy string flat hash map op kernels.
  • Add support for add_metric in the graph function mode.
  • Updating cosine similarity loss — removed the negate sign from cosine similarity.
  • Changed default for gradient accumulation for TPU embeddings to true.
  • Adds summary trace API for collecting graph and profile information.
  • The precision_mode argument to TrtGraphConverter is now case insensitive.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

1e100, a6802739, 4d55397500, a6802739, Abdullah Selek, abenmao, Abolfazl Shahbazi, Adam Richter, Adam Weiss, Ag Ramesh, Alan Du, Albin Joy, Alex, Alex Itkes, Alex Sergeev, Alexander Pivovarov, Alexey Romanov, alhkad, Aman Patel, Amit, Amit Kumar Jaiswal, Amit Srivastava, amoitra, Andreas Eberle, Andrew Lihonosov, Andy Craze, Anshuman Tripathy, Anthony Hsu, Anthony Platanios, Anuj Rawat, arp95, Arpit Shah, Armen Poghosov, armenpoghosov, Astropeak, Ashwin Ramaswami, Arpit Shah, Augustina Ragwitz, Aurelien Geron, AuréLien Geron, avasid, aweers, awesomealex1, Ayush Agrawal, Bas Aarts, Bastian Eichenberger, Bairen Yi, Bayberry Z, Ben Barsdell, Benjamin Peterson, bhack, Bharat Raghunathan, Bhavani Subramanian, Bin Fan, blairhan, BléNesi Attila, Bodin-E, Brandon Carter, Bryan Cutler, candy.dc, Cao Zongyan, Casper Da Costa-Luis, Chao Liu, Chen Guoyin, chenchc, chengchingwen, chie8842, Christian Hansen, Christoph Boeddeker, Christopher Yeh, Clayne Robison, Coady, Patrick, crafet, csukuangfj, ctiijima, Dan Jarvis, Dan Lazewatsky, Daniel Ingram, Daniel Rasmussen, Daniel Salvadori, Dave Airlie, David Norman, Dayananda V, delock, Denis Khalikov, Deven Desai, Dheeraj Rajaram Reddy, Diego Caballero, dmitrievanthony, Donovan Ong, Drew Szurko, Duncan Dean, Duncan Riach, Dustin Neighly, Dwight J Lyle, Eamon Ito-Fisher, eashtian3, Edward Forgacs, EFanZh, ejot, Elroy Ashtian Jr, Eric Schweitz, Evgeniy Polyakov, Fangjun Kuang, Federico Martinez, Fei Hu, Felix Lemke, Filip Matzner, FlashTek, fo40225, formath, FrançOis Chollet, frreiss, Fred Reiss, Frederic Bastien, Fredrik Knutsson, G. Hussain Chinoy, Gabriel, Gautam, gehring, Geoffrey Irving, George Grzegorz Pawelczak, Grzegorz Pawelczak, George Sterpu, Gianluca Varisco, Gleb Popov, Greg Peatfield, Guillaume Klein, Gurpreet Singh, Gustavo Lima Chaves, Gyoung-Yoon Ryoo, haison, Hanton Yang, HanGuo97, Haraldur TóMas HallgríMsson, Hari Shankar, hehongliang, Heungsub Lee, Hoeseong Kim, Huan Li (李卓桓), HåKon Sandsmark, I-Hong, I-Hong Jhuo, Ilham Firdausi Putra, Ilango R, Imran Salam, Innovimax, Jacky Ko, Irene Dea, Ivan Habernal, Jakub Lipinski, Jacky, Jason Zaman, Jason Zavaglia, jayhpark530, jcf94, jefby, Jeff Daily, Jeff Poznanovic, Jeffrey Poznanovic, Jekyll Lai, jer, Jeroen BéDorf, jerryyin, jhalakp, jiakai, Jia Qingtong, Jiankang, JiangXIAO, Joe Bowser, Joe Q, Joe Quadrino, Joel Shapiro, Johan Gunnarsson, Jojimon Varghese, Jonas Rauber, Jonathan Kyl, Jonathan, Joon, Joppe Geluykens, Joseph Friedman, Josh Beal, jtressle, Julian Niedermeier, Junqin Zhang, Justin Dujardin, Justin Tunis, jwu, K. Hodges, kaixih, Kaixi Hou, kjopek, Karl Lessard, Karl Weinmeister, Karthik Muthuraman, Kashif Rasul, Kay Zhu, Kbhute-Ibm, KDR, Keno Fischer, Kevin Mader, khanhlvg, Kilaru Yasaswi Sri Chandra Gandhi, Koan-Sin Tan, Koock Yoon, kouml, ktaebum, Kyuwon Kim, Lakshay Tokas, Laurent Le Brun, leike666666, leonard951, Leslie-Fang, Letian Kang, Li, Guizi, Loo Rong Jie, Lucas Hendren, Lukas Folle, Lukas Geiger, Luke Han, luxupu, lvli, Ma, Guokai, Mahmoud Abuzaina, Maksym Kysylov, Mandar Deshpande, manhyuk, Manraj Singh Grover, Marco Gaido, Marek Drozdowski, Margaret Maynard-Reid, Mark Ryan, mars20, Mateusz Chudyk, Matt Conley, mbhuiyan, mdfaijul, Mei Jie, Melissa Grueter, merturl, MichaelKonobeev, Michael KäUfl, Michal W. Tarnowski, MickaëL Schoentgen, Miguel Morin, Mihail Salnikov, Mikalai Drabovich, Mike Arpaia, Mike Holcomb, minds, monklof, Moses Marin, mpppk, Mr. Metal, Mshr-H, musikisomorphie, nammbash, Natalia Gimelshein, Nathan Luehr, Nayana-Ibm, Nayana Thorat, neargye, Neeraj Pradhan, Nehal J Wani, Neil, Nick, Nick Lewycky, Niels Ole Salscheider, Niklas SilfverströM, Niranjan Hasabnis, Nuka-137, Nutti, ocjosen, olicht, omeir1, P Sudeepam, Paige Bailey, Palmer Lao, Pan Daoxin, Pariksheet Pinjari, Pasquale Minervini, Patrick J. Lopresti, Patrik Gustavsson, Pavel Akhtyamov, Pavel Samolysov, PENGWA, per1234, PeterLee, Phan Van Nguyen Duc, Philipp Jund, Phillip Kravtsov, Pooya Davoodi, Pranav Marathe, Putra Manggala, Qingqing Cao, R S Nikhil Krishna, Rajeshwar Reddy T, Ramon ViñAs, Rasmus Diederichsen, Reuben Morais, robert, Rohit Gupta, Roland Zimmermann, Roman Soldatow, RonLek, Ruizhe, Ryan Jiang, saishruthi, Saleem Abdulrasool, Samantha Andow, Sami Kama, Sami Kama, Sana-Damani, Saurabh Deoras, sdamani, Sean Morgan, seanshpark, Sebastien Iooss, Serv-Inc, Severen Redwood, Shahzad Lone, Shashank Gupta, shashvat, Shashvat Chand Shahi, Shubham Goyal, Shashi, Sigrid Keydana, Siju, Siju Samuel, sleighsoft, smilu97, Snease-Abq, Son Tran, Spencer Schaber, sremedios, Srini511, srinivasan.narayanamoorthy, Steve Lang, Steve Nesae, Subin, Sumesh Udayakumaran, Sungmann Cho, sunway513, Supriya Rao, sxwang, Tae-Hwan Jung, Taehoon Lee, Takeo Sawada, Taylor Jakobson, Taylor Thornton, Ted Chang, TengLu, terryky, ThisIsIsaac, ThisIsPIRI, Thomas Deegan, Thomas Hagebols, tianyapiaozi, Till Hoffmann, Tim Zaman, tomguluson92, Tongxuan Liu, Trent Lo, Trevor Morris, TungJerry, Tyorden, Uday Bondhugula, v1incent, Vagif, Vasileios Lioutas, vbvg2008, vcarpani, Vijay Ravichandran, Vikram Tiwari,Viktor Gal, Vishwak Srinivasan, Vincent, Vishnuvardhan Janapati, Vitor-Alves, Vivek Suryamurthy, wangsiyu, wateryzephyr, WeberXie, Wei Wang, WeijieSun, Wen-Heng (Jack) Chung, wenxizhu, Will Battel, William D. Irons, winstonq, wyzhao, Xiaoming (Jason) Cui, Xiaoquan Kong, Xin, Xinping Wang, Yan Facai (颜发才), Yann-Yy, Yasir Modak, Yasuhiro Matsumoto, ymodak, Yong Tang, Yongfeng Gu, Younes Khoudli, Yuan Lin, Yuan (Terry) Tang, Yuchen Ying, Yves-Noel Weweler, zhangyujing, zjjott, zyeric, 王振华 (Zhenhua Wang), 黄鑫

goldiegadde released this Sep 23, 2020 · 35 commits to r2.0 since this release

Release 2.0.0-rc2

Major Features and Improvements

TensorFlow 2.0 focuses on simplicity and ease of use, featuring updates like:

  • Easy model building with Keras and eager execution.
  • Robust model deployment in production on any platform.
  • Powerful experimentation for research.
  • API simplification by reducing duplication and removing deprecated endpoints.

For details on best practices with 2.0, see the Effective 2.0 guide

For information on upgrading your existing TensorFlow 1.x models, please refer to our Upgrade and Migration guides. We have also released a collection of tutorials and getting started guides.

Highlights

  • TF 2.0 delivers Keras as the central high level API used to build and train models. Keras provides several model-building APIs such as Sequential, Functional, and Subclassing along with eager execution, for immediate iteration and intuitive debugging, and tf.data, for building scalable input pipelines. Checkout guide for additional details.
  • Distribution Strategy: TF 2.0 users will be able to use the tf.distribute.Strategy API to distribute training with minimal code changes, yielding great out-of-the-box performance. It supports distributed training with Keras model.fit, as well as with custom training loops. Multi-GPU support is available, along with experimental support for multi worker and Cloud TPUs. Check out the guide for more details.
  • Functions, not Sessions. The traditional declarative programming model of building a graph and executing it via a tf.Session is discouraged, and replaced by writing regular Python functions. Using the tf.function decorator, such functions can be turned into graphs which can be executed remotely, serialized, and optimized for performance.
  • Unification of tf.train.Optimizers and tf.keras.Optimizers. Use tf.keras.Optimizers for TF2.0. compute_gradients is removed as public API, and use GradientTape to compute gradients.
  • AutoGraph translates Python control flow into TensorFlow expressions, allowing users to write regular Python inside tf.function -decorated functions. AutoGraph is also applied in functions used with tf.data, tf.distribute and tf.keras APIs.
  • Unification of exchange formats to SavedModel. All TensorFlow ecosystem projects (TensorFlow Lite, TensorFlow JS, TensorFlow Serving, TensorFlow Hub) accept SavedModels. Model state should be saved to and restored from SavedModels.
  • API Changes: Many API symbols have been renamed or removed, and argument names have changed. Many of these changes are motivated by consistency and clarity. The 1.x API remains available in the compat.v1 module. A list of all symbol changes can be found here.
  • API clean-up, included removing tf.app , tf.flags , and tf.logging in favor of absl-py.
  • No more global variables with helper methods like tf.global_variables_initializer and tf.get_global_step .
  • Add toggles tf.enable_control_flow_v2() and tf.disable_control_flow_v2() for enabling/disabling v2 control flow.
  • Enable v2 control flow as part of tf.enable_v2_behavior() and TF2_BEHAVIOR=1 .
  • Fixes autocomplete for most TensorFlow API references by switching to use relative imports in API __init__.py files.
  • Auto Mixed-Precision graph optimizer simplifies converting models to float16 for acceleration on Volta and Turing Tensor Cores. This feature can be enabled by wrapping an optimizer class with tf.train.experimental.enable_mixed_precision_graph_rewrite() .
  • Add environment variable TF_CUDNN_DETERMINISTIC . Setting to «true» or «1» forces the selection of deterministic cuDNN convolution and max-pooling algorithms. When this is enabled, the algorithm selection procedure itself is also deterministic.

Breaking Changes

Many backwards incompatible API changes have been made to clean up the APIs and make them more consistent.

  • TensorFlow 2.0.0 is built using devtoolset7 (GCC7) on Ubuntu 16. This may lead to ABI incompatibilities with extensions built against earlier versions of TensorFlow.
  • Tensorflow code now produces 2 different pip packages: tensorflow_core containing all the code (in the future it will contain only the private implementation) and tensorflow which is a virtual pip package doing forwarding to tensorflow_core (and in the future will contain only the public API of tensorflow). We don’t expect this to be breaking, unless you were importing directly from the implementation.
  • tf.contrib has been deprecated, and functionality has been either migrated to the core TensorFlow API, to an ecosystem project such as tensorflow/addons or tensorflow/io, or removed entirely.
  • Remove tf.contrib.timeseries dependency on TF distributions.
  • Replace contrib references with tf.estimator.experimental.* for apis in early_stopping.py.
  • Premade estimators in the tf.estimator.DNN/Linear/DNNLinearCombined family have been updated to use tf.keras.optimizers instead of the tf.compat.v1.train.Optimizer s. If you do not pass in an optimizer= arg or if you use a string, the premade estimator will use the Keras optimizer. This is checkpoint breaking, as the optimizers have separate variables. A checkpoint converter tool for converting optimizers is included with the release, but if you want to avoid any change, switch to the v1 version of the estimator: tf.compat.v1.estimator.DNN/Linear/DNNLinearCombined* .
  • Default aggregation for canned Estimators is now SUM_OVER_BATCH_SIZE . To maintain previous default behavior, please pass SUM as the loss aggregation method.
  • Canned Estimators don’t support input_layer_partitioner arg in the API. If you have this arg, you will have to switch to tf.compat.v1 canned Estimators .
  • Estimator.export_savedmodel has been renamed export_saved_model .
  • When saving to SavedModel, Estimators will strip default op attributes. This is almost always the correct behavior, as it is more forwards compatible, but if you require that default attributes are saved with the model, please use tf.compat.v1.Estimator .
  • Feature Columns have been upgraded to be more Eager-friendly and to work with Keras. As a result, tf.feature_column.input_layer has been deprecated in favor of tf.keras.layers.DenseFeatures . v1 feature columns have direct analogues in v2 except for shared_embedding_columns , which are not cross-compatible with v1 and v2. Use tf.feature_column.shared_embeddings instead.
  • OMP_NUM_THREADS is no longer used by the default Keras config. To configure the number of threads, use tf.config.threading APIs.
  • tf.keras.model.save_model and model.save now defaults to saving a TensorFlow SavedModel. HDF5 files are still supported.
  • Deprecated tf.keras.experimental.export_saved_model and tf.keras.experimental.function . Please use tf.keras.models.save_model(. save_format=’tf’) and tf.keras.models.load_model instead.
  • Layers now default to float32, and automatically cast their inputs to the layer’s dtype. If you had a model that used float64, it will probably silently use float32 in TensorFlow 2, and a warning will be issued that starts with «Layer is casting an input tensor from dtype float64 to the layer’s dtype of float32». To fix, either set the default dtype to float64 with tf.keras.backend.set_floatx(‘float64′) , or pass dtype=’float64’ to each of the Layer constructors. See tf.keras.layers.Layer for more information.
  • Removed lite.OpHint , lite.experimental , and lite.constant from 2.0 API.

Tensors are no longer hashable, but instead compare element-wise with == and != . Use tf.compat.v1.disable_tensor_equality() to return to the previous behavior.

Performing equality operations on Tensors or Variables with incompatible shapes an exception is no longer thrown. Instead __eq__ returns False and __ne__ returns True.

Removed tf.string_split from v2 API.

Deprecated the use of constraint= and .constraint with ResourceVariable.

Add UnifiedGRU as the new GRU implementation for tf2.0. Change the default recurrent activation function for GRU from hard_sigmoid to sigmoid , and reset_after to True in 2.0. Historically recurrent activation is hard_sigmoid since it is fast than ‘sigmo , reset_after=False) to fallback to 1.x behavior.

CUDNN_INSTALL_PATH , TENSORRT_INSTALL_PATH , NCCL_INSTALL_PATH , NCCL_HDR_PATH are deprecated. Use TF_CUDA_PATHS instead which supports a comma-separated list of base paths that are searched to find CUDA libraries and headers.

Refer to our public project status tracker and issues tagged with 2.0 on GitHub for insight into recent issues and development progress.

If you experience any snags when using TF 2.0, please let us know at the TF 2.0 Testing User Group. We have a support mailing list as well as weekly testing meetings, and would love to hear your migration feedback and questions.

Bug Fixes and Other Changes

  • Expose tf.contrib.proto.* ops in tf.io (they will exist in TF2)
  • Add support for TensorArrays to tf.data Dataset .
  • Integrate Ragged Tensors with tf.data .
  • All core and experimental tf.data transformations that input user-defined functions can span multiple devices now.
  • Extending the TF 2.0 support for shuffle(. reshuffle_each_iteration=True) and cache() to work across different Python iterators for the same dataset.
  • Removing the experimental_numa_aware option from tf.data.Options .
  • Add num_parallel_reads and passing in a Dataset containing filenames into TextLineDataset and FixedLengthRecordDataset .
  • Add support for defaulting the value of cycle_length argument of tf.data.Dataset.interleave to the number of schedulable CPU cores.
  • Promoting tf.data.experimental.enumerate_dataset to core as tf.data.Dataset.enumerate .
  • Promoting tf.data.experimental.unbatch to core as tf.data.Dataset.unbatch .
  • Adds option for introducing slack in the pipeline to reduce CPU contention, via tf.data.Options().experimental_slack = True
  • Added experimental support for parallel batching to batch() and padded_batch() . This functionality can be enabled through tf.data.Options()
  • Support cancellation of long-running reduce .
  • Now we use dataset node name as prefix instead of the op name, to identify the component correctly in metrics, for pipelines with repeated components.
  • Improve the performance of datasets using from_tensors() .
  • Promoting unbatch from experimental to core API.
  • Adding support for datasets as inputs to from_tensors and from_tensor_slices and batching and unbatching of nested datasets.
  • Enable tf.distribute.experimental.MultiWorkerMirroredStrategy working in eager mode.
  • Callbacks are supported in MultiWorkerMirroredStrategy .
  • Disable run_eagerly and distribution strategy if there are symbolic tensors added to the model using add_metric or add_loss .
  • Loss and gradients should now more reliably be correctly scaled w.r.t. the global batch size when using a tf.distribute.Strategy .
  • Set default loss reduction as AUTO for improving reliability of loss scaling with distribution strategy and custom training loops. AUTO indicates that the reduction option will be determined by the usage context. For almost all cases this defaults to SUM_OVER_BATCH_SIZE . When used in distribution strategy scope, outside of built-in training loops such as tf.keras compile and fit , we expect reduction value to be ‘None’ or ‘SUM’. Using other values will raise an error.
  • Support for multi-host ncclAllReduce in Distribution Strategy.
  • Replace tf.contrib.estimator.add_metrics with tf.estimator.add_metrics
  • Use tf.compat.v1.estimator.inputs instead of tf.estimator.inputs
  • Replace contrib references with tf.estimator.experimental.* for apis in early_s in Estimator
  • Canned Estimators will now use keras optimizers by default. An error will be raised if tf.train.Optimizers are used, and you will have to switch to tf.keras.optimizers or tf.compat.v1 canned Estimators.
  • A checkpoint converter for canned Estimators has been provided to transition canned Estimators that are warm started from tf.train.Optimizers to tf.keras.optimizers .
  • Losses are scaled in canned estimator v2 and not in the optimizers anymore. If you are using Estimator + distribution strategy + optimikzer v1 then the behavior does not change. This implies that if you are using custom estimator with optimizer v2, you have to scale losses. We have new utilities to help scale losses tf.nn.compute_average_loss , tf.nn.scale_regularization_loss .
  • Premade models (including Linear and WideDeep) have been introduced for the purpose of replacing Premade estimators.
  • Model saving changes
  • model.save and tf.saved_model.save may now save to the TensorFlow SavedModel format. The model can be restored using tf.keras.models.load_model . HDF5 files are still supported, and may be used by specifying save_format=»h5″ when saving.
  • Raw TensorFlow functions can now be used in conjunction with the Keras Functional API during model creation. This obviates the need for users to create Lambda layers in most cases when using the Functional API. Like Lambda layers, TensorFlow functions that result in Variable creation or assign ops are not supported.
  • Add support for passing list of lists to the metrics argument in Keras compile .
  • Add tf.keras.layers.AbstractRNNCell as the preferred implementation for RNN cells in TF v2. User can use it to implement RNN cells with custom behavior.
  • Keras training and validation curves are shown on the same plot when using the TensorBoard callback.
  • Switched Keras fit/evaluate/predict execution to use only a single unified path by default unless eager execution has been explicitly disabled, regardless of input type. This unified path places an eager-friendly training step inside of a tf.function . With this
    1. All input types are converted to Dataset .
    2. The path assumes there is always a distribution strategy, when distribution strategy is not specified the path uses a no-op distribution strategy.
    3. The training step is wrapped in tf.function unless run_eagerly=True is set in compile. The single path execution code does not yet support all use cases. We fallback to the existing v1 execution paths if your model contains the following:
      1. sample_weight_mode in compile
      2. weighted_metrics in compile
      3. v1 optimizer
      4. target tensors in compile.
    4. If you are experiencing any issues because of this change, please inform us (file an issue) about your use case and you can unblock yourself by setting experimental_run_tf_function=False in compile meanwhile. We have seen couple of use cases where the model usage pattern is not as expected and would not work with this change.
      1. output tensors of one layer is used in the constructor of another.
      2. symbolic tensors outside the scope of the model are used in custom loss functions.
        The flag can be disabled for these cases and ideally the usage pattern will need to be fixed.
  • Mark Keras set_session as compat.v1 only.
  • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format , which allows the saved checkpoints to be compatible with model.load_weights .
  • keras.backend.resize_images (and consequently, keras.layers.Upsampling2D ) behavior has changed, a bug in the resizing implementation was fixed.
  • Add an implementation=3 mode for tf.keras.layers.LocallyConnected2D and tf.keras.layers.LocallyConnected1D layers using tf.SparseTensor to store weights, allowing a dramatic speedup for large sparse models.
  • Raise error if batch_size argument is used when input is dataset/generator/keras sequence.
  • Update TF 2.0 keras.backend.name_scope to use TF 2.0 name_scope .
  • Add v2 module aliases for losses, metrics, initializers and optimizers: tf.losses = tf.keras.losses & tf.metrics = tf.keras.metrics & tf.initializers = tf.keras.initializers & tf.optimizers = tf.keras.optimizers .
  • Updates binary cross entropy logic in Keras when input is probabilities. Instead of converting probabilities to logits, we are using the cross entropy formula for probabilities.
  • Added public APIs for cumsum and cumprod keras backend functions.
  • Add support for temporal sample weight mode in subclassed models.
  • Raise ValueError if an integer is passed to the training APIs.
  • Added fault-tolerance support for training Keras model via model.fit() with MultiWorkerMirroredStrategy , tutorial available.
  • Custom Callback tutorial is now available.
  • To train with tf.distribute , Keras api is recommended over estimator.
  • steps_per_epoch and steps arguments are supported with numpy arrays.
  • New error message when unexpected keys are used in sample_weight/class_weight dictionaries
  • Losses are scaled in Keras compile/fit and not in the optimizers anymore. If you are using custom training loop, we have new utilities to help scale losses tf.nn.compute_average_loss , tf.nn.scale_regularization_loss .
  • Layer apply and add_variable APIs are deprecated.
  • Added support for channels first data format in cross entropy losses with logits and support for tensors with unknown ranks.
  • Error messages will be raised if add_update , add_metric , add_loss , activity regularizers are used inside of a control flow branch.
  • New loss reduction types:
    1. AUTO : Indicates that the reduction option will be determined by the usage context. For almost all cases this defaults to SUM_OVER_BATCH_SIZE . When used with tf.distribute.Strategy , outside of built-in training loops such as tf.keras compile and fit , we expect reduction value to be SUM or NONE . Using AUTO in that case will raise an error.
    2. NONE : Weighted losses with one dimension reduced (axis=-1, or axis specified by loss function). When this reduction type used with built-in Keras training loops like fit / evaluate , the unreduced vector loss is passed to the optimizer but the reported loss will be a scalar value.
    3. SUM : Scalar sum of weighted losses. 4. SUM_OVER_BATCH_SIZE : Scalar SUM divided by number of elements in losses. This reduction type is not supported when used with tf.distribute.Strategy outside of built-in training loops like tf.keras compile / fit .
  • Wraps losses passed to the compile API (strings and v1 losses) which are not instances of v2 Loss class in LossWrapper class. => All losses will now use SUM_OVER_BATCH_SIZE reduction as default.
  • model.add_loss(symbolic_tensor) should work in ambient eager.
  • Update metric name to always reflect what the user has given in compile. Affects following cases
    1. When name is given as ‘accuracy’/’crossentropy’
    2. When an aliased function name is used eg. ‘mse’
    3. Removing the weighted prefix from weighted metric names.
  • Allow non-Tensors through v2 losses.
  • Add v2 sparse categorical crossentropy metric.
  • Add v2 APIs for AUCCurve and AUCSummationMethod enums.
  • add_update can now be passed a zero-arg callable in order to support turning off the update when setting trainable=False on a Layer of a Model compiled with run_eagerly=True .
  • Standardize the LayerNormalization API by replacing the args norm_axis and params_axis with axis .
  • Fixed critical bugs that help with DenseFeatures usability in TF2
  • Added evaluation script for COCO minival
  • Add delegate support for QUANTIZE .
  • Add GATHER support to NN API delegate.
  • Added support for TFLiteConverter Python API in 2.0. Contains functions from_saved_model, from_keras_file, and from_concrete_functions.
  • Add EXPAND_DIMS support to NN API delegate TEST.
  • Add narrow_range attribute to QuantizeAndDequantizeV2 and V3.
  • Added support for tflite_convert command line tool in 2.0.
  • Post-training quantization tool supports quantizing weights shared by multiple operations. The models made with versions of this tool will use INT8 types for weights and will only be executable interpreters from this version onwards.
  • Post-training quantization tool supports fp16 weights and GPU delegate acceleration for fp16.
  • Add delegate support for QUANTIZED_16BIT_LSTM .
  • Extracts NNAPIDelegateKernel from nnapi_delegate.cc
  • Add TensorFlow 2.0-compatible TrtGraphConverterV2 API for TensorRT conversion. TensorRT initialization arguments are now passed wrapped in a named-tuple, TrtConversionParams , rather than as separate arguments as in TrtGraphConverter .
  • Changed API to optimize TensorRT engines during graph optimization. This is nowdone by calling converter.build() where previously is_dynamic_op=False would be set.
  • converter.convert() no longer returns a tf.function . Now the function must beaccessed from the saved model.
  • The converter.calibrate() method has been removed. To trigger calibration, a calibration_input_fn should be provided to converter.convert() .
  • Fix accidental quadratic graph construction cost in graph-mode tf.gradients() .
  • ResourceVariable’s gather op supports batch dimensions.
  • ResourceVariable support for gather_nd .
  • ResourceVariable and Variable no longer accepts constraint in the constructor, nor expose it as a @Property.
  • Added gradient for SparseToDense op.
  • Expose a flag that allows the number of threads to vary across Python benchmarks.
  • image.resize in 2.0 now supports gradients for the new resize kernels.
  • image.resize now considers proper pixel centers and has new kernels (incl. anti-aliasing).
  • Renamed tf.image functions to remove duplicate «image» where it is redundant.
  • Variadic reduce is supported on CPU Variadic reduce is supported on CPU
  • Remove unused StringViewVariantWrapper .
  • Delete unused Fingerprint64Map op registration
  • Add broadcasting support to tf.matmul .
  • Add C++ Gradient for BatchMatMulV2 .
  • Add tf.math.cumulative_logsumexp operation.
  • Add ellipsis (. ) support for tf.einsum() .
  • Add expand_composites argument to all nest.* methods.
  • Added strings.byte_split .
  • Add a new «result_type» parameter to tf.strings.split .
  • Add name argument to tf.string_split and tf.strings_split .
  • Extend tf.strings.split to support inputs with any rank.
  • Added tf.random.binomial .
  • Added key and skip methods to random.experimental.Generator .
  • Extend tf.function with basic support for CompositeTensors arguments (such as SparseTensor and RaggedTensor ).
  • parallel_for.pfor : add converters for Softmax, LogSoftmax, IsNaN, All, Any, and MatrixSetDiag.
  • parallel_for : add converters for LowerTriangularSolve and Cholesky.
  • parallel_for : add converters for LogMatrixDeterminant and MatrixBandPart .
  • parallel_for : Add converter for MatrixDiag .
  • parallel_for : Add converters for OneHot , LowerBound , UpperBound .
  • parallel_for : add converter for BroadcastTo .
  • Add pfor converter for Squeeze .
  • Add RaggedTensor.placeholder() .
  • Add ragged tensor support to tf.squeeze .
  • Update RaggedTensors to support int32 row_splits.
  • Allow LinearOperator.solve to take a LinearOperator .
  • Allow all dtypes for LinearOperatorCirculant .
  • Introduce MaxParallelism method
  • Add LinearOperatorHouseholder .
  • Adds Philox support to new stateful RNG’s XLA path.
  • Add TensorSpec support for CompositeTensors.
  • Add tf.linalg.tridiagonal_solve op.
  • Added partial_pivoting input parameter to tf.linalg.tridiagonal_solve .
  • Added gradient to tf.linalg.tridiagonal_solve .
  • Add tf.linalg.tridiagonal_mul op .
  • Added GPU implementation of tf.linalg.tridiagonal_matmul .
  • Add LinearOperatorToeplitz .
  • Upgraded LIBXSMM to version 1.11.
  • Uniform processing of quantized embeddings by Gather and EmbeddingLookup Ops
  • Correct a misstatement in the documentation of the sparse softmax cross entropy logit parameter.
  • Add tf.ragged.boolean_mask .
  • tf.switch_case added, which selects a branch_fn based on a branch_index.
  • The C++ kernel of gather op supports batch dimensions.
  • Fixed default value and documentation for trainable arg of tf.Variable.
  • EagerTensor now supports numpy buffer interface for tensors.
  • This change bumps the version number of the FullyConnected Op to 5.
  • Added new op: tf.strings.unsorted_segment_join .
  • Add HW acceleration support for topK_v2 .
  • CloudBigtable version updated to v0.10.0 BEGIN_PUBLIC CloudBigtable version updated to v0.10.0.
  • Expose Head as public API.
  • Added tf.sparse.from_dense utility function.
  • Improved ragged tensor support in TensorFlowTestCase .
  • Added a function nested_value_rowids for ragged tensors.
  • Add tf.ragged.stack .
  • Makes the a-normal form transformation in Pyct configurable as to which nodes are converted to variables and which are not.
  • ResizeInputTensor now works for all delegates.
  • tf.cond emits a StatelessIf op if the branch functions are stateless and do not touch any resources.
  • Add support of local soft device placement for eager op.
  • Pass partial_pivoting to the _TridiagonalSolveGrad .
  • Add HW acceleration support for LogSoftMax .
  • Add guard to avo >AddNewInputConstantTensor .
  • Delegate application failure leaves interpreter in valid state
  • tf.while_loop emits a StatelessWhile op if the cond and body functions are stateless and do not touch any resources.
  • tf.cond, tf.while and if and while in AutoGraph now accept a nonscalar predicate if has a single element. This does not affec non-V2 control flow.
  • Fix potential security vulnerability where decoding variant tensors from proto could result in heap out of bounds memory access.
  • Only create a GCS directory object if the object does not already exist.
  • Introduce dynamic constructor argument in Layer and Model, which should be set to True when using imperative control flow in the call method.
  • Begin adding Go wrapper for C Eager API.
  • XLA HLO graphs can be inspected with interactive_graphviz tool now.
  • Add dataset ops to the graph (or create kernels in Eager execution) during the python Dataset object creation instead doing it during Iterator creation time.
  • Add batch_dims argument to tf.gather .
  • The behavior of tf.gather is now correct when axis=None and batch_dims batch_dims case.
  • Removing of dtype in the constructor of initializers and partition_info in call.
  • Add tf.math.nextafter op.
  • Turn on MKL-DNN contraction kernels by default. MKL-DNN dynamically dispatches the best kernel implementation based on CPU vector architecture. To disable them, build with —define=tensorflow_mkldnn_contraction_kernel=0 .
  • tf.linspace(start, stop, num) now always uses «stop» as last value (for num > 1)
  • Added top-k to precision and recall to keras metrics.
  • Add a ragged size op and register it to the op dispatcher
  • Transitive dependencies on : pooling_ops were removed. Some users may need to add explicit dependencies on : pooling_ops if they reference the operators from that library.
  • Add CompositeTensor base class.
  • Malformed gif images could result in an access out of bounds in the color palette of the frame. This has been fixed now
  • Add templates and interfaces for creating lookup tables
  • Tensor::UnsafeCopyFromInternal deprecated in favor Tensor::BitcastFrom .
  • In map_vectorization optimization, reduce the degree of parallelism in the vectorized map node.
  • Add variant wrapper for absl::string_view .
  • Add OpKernels for some stateless maps.
  • DType is no longer convertible to an int. Use dtype.as_datatype_enum instead of int(dtype) to get the same result.
  • Support both binary and -1/1 label input in v2 hinge and squared hinge losses.
  • Added LinearOperator.adjoint and LinearOperator.H (alias).
  • Expose CriticalSection in core as tf.CriticalSection .
  • Enhanced graphviz output.
  • Add opkernel templates for common table operations.
  • Fix callbacks do not log values in eager mode when a deferred build model is used.
  • SignatureDef util functions have been deprecated.
  • Update Fingerprint64Map to use aliases
  • Add legacy string flat hash map op kernels.
  • Add support for add_metric in the graph function mode.
  • Updating cosine similarity loss — removed the negate sign from cosine similarity.
  • Changed default for gradient accumulation for TPU embeddings to true.
  • Adds summary trace API for collecting graph and profile information.
Мастер Йода рекомендует:  Контент менеджер - новая профессия


Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

1e100, a6802739, Abolfazl Shahbazi, Adam Weiss, Ag Ramesh, Alan Du, Albin Joy, Amit, Amit Srivastava, Andy Craze, Anshuman Tripathy, Armen Poghosov, armenpoghosov, Arpit Shah, Ashwin Ramaswami, Aurelien Geron, AuréLien Geron, aweers, awesomealex1, Bairen Yi, Ben Barsdell, Bhavani Subramanian, Brandon Carter, candy.dc, Chao Liu, Clayne Robison, csukuangfj, Dan Jarvis, Dan Lazewatsky, Daniel Ingram, Dave Airlie, David Norman, Dayananda V, Denis Khalikov, Deven Desai, Dheeraj Rajaram Reddy, dmitrievanthony, Drew Szurko, Duncan Riach, Fei Hu, Felix Lemke, Filip Matzner, fo40225, frreiss, Gautam, gehring, Grzegorz George Pawelczak, Grzegorz Pawelczak, HanGuo97, Hari Shankar, hehongliang, Heungsub Lee, Hoeseong Kim, I-Hong Jhuo, Ilango R, Innovimax, Jacky Ko, Jakub Lipinski, jcf94, Jeff Poznanovic, Jia Qingtong, Jiankang, Joe Q, Joe Quadrino, Jonas Rauber, Jonathan Kyl, Joppe Geluykens, Joseph Friedman, jtressle, jwu, K. Hodges, kaixih, Karl Lessard, Karl Weinmeister, Kashif Rasul, kjopek, Koan-Sin Tan, kouml, ktaebum, Laurent Le Brun, Li, Guizi, Loo Rong Jie, Lucas Hendren, Lukas Geiger, Luke Han, Mahmoud Abuzaina, manhyuk, Marco Gaido, Marek Drozdowski, Mark Ryan, mars20, Mateusz Chudyk, Matt Conley, MattConley, mbhuiyan, mdfaijul, Melissa Grueter, Michael KäUfl, MickaëL Schoentgen, Miguel Morin, Mike Arpaia, nammbash, Natalia Gimelshein, Nayana-Ibm, neargye, Nehal J Wani, Niels Ole Salscheider, Niranjan Hasabnis, Nutti, olicht, P Sudeepam, Paige Bailey, Palmer Lao, Pariksheet Pinjari, Pavel Samolysov, Pooya Davoodi, Ryan Jiang, Samantha Andow, Sami Kama, Saurabh Deoras, Shahzad Lone, Shashi, Siju, Siju Samuel, Snease-Abq, Spencer Schaber, srinivasan.narayanamoorthy, Steve Lang, Steve Nesae, Supriya Rao, Taylor Jakobson, Taylor Thornton, ThisIsPIRI, Thomas Deegan, tomguluson92, Tongxuan Liu, Vagif, vcarpani, Vikram Tiwari, Vishwak Srinivasan, Vitor-Alves, wangsiyu, WeberXie, WeijieSun, Wen-Heng (Jack) Chung, wenxizhu, William D. Irons, Yan Facai (颜发才), ymodak, Younes Khoudli, Yuan Lin, Yves-Noel Weweler, zjjott, 卜居, 王振华 (Wang Zhenhua),

Мастер Йода рекомендует:  Встречают по одежке или используем английский в IT

4d55397500, a6802739, Abdullah Selek, abenmao, Adam Richter, Ag Ramesh, Albin Joy, Alex, Alex Itkes, Alex Sergeev, Alexander Pivovarov, Alexey Romanov, alhkad, Aman Patel, Amit, Amit Kumar Jaiswal, Amit Srivastava, amoitra, Andreas Eberle, Andrew Lihonosov, Anthony Hsu, Anthony Platanios, Anuj Rawat, arp95, Arpit Shah, Astropeak, Augustina Ragwitz, Aurelien Geron, AuréLien Geron, avasid, aweers, Ayush Agrawal, Bas Aarts, Bastian Eichenberger, Bayberry Z, Ben Barsdell, Benjamin Peterson, bhack, Bharat Raghunathan, Bhavani Subramanian, Bin Fan, blairhan, BléNesi Attila, Bodin-E, Bryan Cutler, Cao Zongyan, Casper Da Costa-Luis, Chen Guoyin, chenchc, chengchingwen, chie8842, Christian Hansen, Christoph Boeddeker, Christopher Yeh, Clayne Robison, Coady, Patrick, crafet, ctiijima, Daniel Rasmussen, Daniel Salvadori, David Norman, delock, Denis Khalikov, Deven Desai, Diego Caballero, Donovan Ong, Duncan Dean, Duncan Riach, Dustin Neighly, Dwight J Lyle, Eamon Ito-Fisher, eashtian3, Edward Forgacs, EFanZh, ejot, Elroy Ashtian Jr, Eric Schweitz, Evgeniy Polyakov, Fangjun Kuang, Federico Martinez, Fei Hu, Filip Matzner, FlashTek, fo40225, formath, FrançOis Chollet, Fred Reiss, Frederic Bastien, Fredrik Knutsson, G. Hussain Chinoy, Gabriel, gehring, Geoffrey Irving, George Grzegorz Pawelczak, George Sterpu, Gianluca Varisco, Gleb Popov, Greg Peatfield, Guillaume Klein, Gurpreet Singh, Gustavo Lima Chaves, Gyoung-Yoon Ryoo, haison, Hanton Yang, Haraldur TóMas HallgríMsson, Huan Li (李卓桓), HåKon Sandsmark, I-Hong, Ilham Firdausi Putra, Imran Salam, Irene Dea, Ivan Habernal, Jacky, Jason Zaman, Jason Zavaglia, jayhpark530, jefby, Jeff Daily, Jeffrey Poznanovic, Jekyll Lai, jer, Jeroen BéDorf, jerryyin, jhalakp, jiakai, JiangXIAO, Joe Bowser, Joel Shapiro, Johan Gunnarsson, Jojimon Varghese, Jonathan, Joon, Josh Beal, Julian Niedermeier, Junqin Zhang, Justin Dujardin, Justin Tunis, Kaixi Hou, Karthik Muthuraman, Kay Zhu, Kbhute-Ibm, KDR, Keno Fischer, Kevin Mader, khanhlvg, Kilaru Yasaswi Sri Chandra Gandhi, Koock Yoon, Kyuwon Kim, Lakshay Tokas, leike666666, leonard951, Leslie-Fang, Letian Kang, Li, Guizi, Lukas Folle, Lukas Geiger, luxupu, lvli, Ma, Guokai, Mahmoud Abuzaina, Maksym Kysylov, Mandar Deshpande, Manraj Singh Grover, Margaret Maynard-Reid, Mark Ryan, Matt Conley, mbhuiyan, mdfaijul, Mei Jie, merturl, MichaelKonobeev, Michal W. Tarnowski, Mihail Salnikov, Mikalai Drabovich, Mike Holcomb, minds, monklof, Moses Marin, mpppk, Mr. Metal, Mshr-H, musikisomorphie, nammbash, Nathan Luehr, Nayana Thorat, Neeraj Pradhan, Neil, Nick, Nick Lewycky, Niels Ole Salscheider, Niklas SilfverströM, Niranjan Hasabnis, Nuka-137, ocjosen, omeir1, P Sudeepam, Pan Daoxin, Pariksheet Pinjari, Pasquale Minervini, Patrick J. Lopresti, Patrik Gustavsson, Pavel Akhtyamov, PENGWA, per1234, PeterLee, Phan Van Nguyen Duc, Philipp Jund, Phillip Kravtsov, Pooya Davoodi, Pranav Marathe, Putra Manggala, Qingqing Cao, R S Nikhil Krishna, Rajeshwar Reddy T, Ramon ViñAs, Rasmus Diederichsen, Reuben Morais, robert, Rohit Gupta, Roland Zimmermann, Roman Soldatow, RonLek, Ruizhe, Ryan Jiang, saishruthi, Saleem Abdulrasool, Sami Kama, Sana-Damani, sdamani, Sean Morgan, seanshpark, Sebastien Iooss, Serv-Inc, Severen Redwood, Shashank Gupta, shashvat, Shashvat Chand Shahi, Shubham Goyal, Sigrid Keydana, Siju Samuel, sleighsoft, smilu97, Son Tran, sremedios, Srini511, srinivasan.narayanamoorthy, Subin, Sumesh Udayakumaran, Sungmann Cho, sunway513, sxwang, Tae-Hwan Jung, Taehoon Lee, Takeo Sawada, Taylor Jakobson, Ted Chang, TengLu, terryky, ThisIsIsaac, Thomas Deegan, Thomas Hagebols, tianyapiaozi, Till Hoffmann, Tim Zaman, Tongxuan Liu, Trent Lo, Trevor Morris, TungJerry, Tyorden, Uday Bondhugula, v1incent, Vasileios Lioutas, vbvg2008, Vijay Ravichandran, Viktor Gal, Vincent, Vishnuvardhan Janapati, Vivek Suryamurthy, wangsiyu, wateryzephyr, Wei Wang, Wen-Heng (Jack) Chung, wenxizhu, Will Battel, William D. Irons, winstonq, wyzhao, Xiaoming (Jason) Cui, Xiaoquan Kong, Xin, Xinping Wang, Yann-Yy, Yasir Modak, Yasuhiro Matsumoto, Yong Tang, Yongfeng Gu, Yuan (Terry) Tang, Yuchen Ying, zhangyujing, zyeric, 王振华 (Zhenhua Wang), 黄鑫

goldiegadde released this Sep 17, 2020 · 74 commits to r1.15 since this release

Release 1.15.0-rc1

This is the last 1.x release for TensorFlow. We do not expect to update the 1.x branch with features, although we will issue patch releases to fix vulnerabilities for at least one year.

Major Features and Improvements

  • As announced, tensorflow pip package will by default include GPU support (same as tensorflow-gpu now) for the platforms we currently have GPU support (Linux and Windows). It will work on machines with and without Nvidia GPUs. tensorflow-gpu will still be available, and CPU-only packages can be downloaded at tensorflow-cpu for users who are concerned about package size.
  • TensorFlow 1.15 contains a complete implementation of the 2.0 API in its compat.v2 module . It contains a copy of the 1.15 main module (without contrib ) in the compat.v1 module . TensorFlow 1.15 is able to emulate 2.0 behavior using the enable_v2_behavior() function.
    This enables writing forward compatible code: by explicitly importing either tensorflow.compat.v1 or tensorflow.compat.v2 , you can ensure that your code works without modifications against an installation of 1.15 or 2.0.
  • EagerTensor now supports numpy buffer interface for tensors.
  • Add toggles tf.enable_control_flow_v2() and tf.disable_control_flow_v2() for enabling/disabling v2 control flow.
  • Enable v2 control flow as part of tf.enable_v2_behavior() and TF2_BEHAVIOR=1 .
  • AutoGraph translates Python control flow into TensorFlow expressions, allowing users to write regular Python inside tf.function -decorated functions. AutoGraph is also applied in functions used with tf.data , tf.distribute and tf.keras APIS.
  • Adds enable_tensor_equality() , which switches the behavior such that:
    • Tensors are no longer hashable.
    • Tensors can be compared with == and !=, yielding a Boolean Tensor with element-wise comparison results. This will be the default behavior in 2.0.
  • Auto Mixed-Precision graph optimizer simplifies converting models to float16 for acceleration on Volta and Turing Tensor Cores. This feature can be enabled by wrapping an optimizer class with tf.train.experimental.enable_mixed_precision_graph_rewrite() .
  • Add environment variable TF_CUDNN_DETERMINISTIC . Setting to «true» or «1» forces the selection of deterministic cuDNN convolution and max-pooling algorithms. When this is enabled, the algorithm selection procedure itself is also deterministic.
  • TensorRT
    • Migrate TensorRT conversion sources from contrib to compiler directory in preparation for TF 2.0.
    • Add additional, user friendly TrtGraphConverter API for TensorRT conversion.
    • Expand support for TensorFlow operators in TensorRT conversion (e.g.
      Gather , Slice , Pack , Unpack , ArgMin , ArgMax , DepthSpaceShuffle ).
    • Support TensorFlow operator CombinedNonMaxSuppression in TensorRT conversion which
      significantly accelerates object detection models.

Breaking Changes

  • Tensorflow code now produces 2 different pip packages: tensorflow_core containing all the code (in the future it will contain only the private implementation) and tensorflow which is a virtual pip package doing forwarding to tensorflow_core (and in the future will contain only the public API of tensorflow). We don’t expect this to be breaking, unless you were importing directly from the implementation.
  • TensorFlow 1.15 is built using devtoolset7 (GCC7) on Ubuntu 16. This may lead to ABI incompatibilities with extensions built against earlier versions of TensorFlow.
  • Deprecated the use of constraint= and .constraint with ResourceVariable.
  • tf.keras :
    • OMP_NUM_THREADS is no longer used by the default Keras config. To configure the number of threads, use tf.config.threading APIs.
    • tf.keras.model.save_model and model.save now defaults to saving a TensorFlow SavedModel.
    • keras.backend.resize_images (and consequently, keras.layers.Upsampling2D ) behavior has changed, a bug in the resizing implementation was fixed.
    • Layers now default to float32 , and automatically cast their inputs to the layer’s dtype. If you had a model that used float64 , it will probably silently use float32 in TensorFlow2, and a warning will be issued that starts with Layer «layer-name» is casting an input tensor from dtype float64 to the layer’s dtype of float32. To fix, either set the default dtype to float64 with tf.keras.backend.set_floatx(‘float64′) , or pass dtype=’float64’ to each of the Layer constructors. See tf.keras.layers.Layer for more information.
    • Some tf.assert_* methods now raise assertions at operation creation time (i.e. when this Python line executes) if the input tensors’ values are known at that time, not during the session.run(). When this happens, a noop is returned and the input tensors are marked non-feedable. In other words, if they are used as keys in feed_dict argument to session.run() , an error will be raised. Also, because some assert ops don’t make it into the graph, the graph structure changes. A different graph can result in different per-op random seeds when they are not given explicitly (most often).

Bug Fixes and Other Changes

  • tf.estimator :
    • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format, which allows the saved checkpoints to be compatible with model.load_weights .
    • Fix tests in canned estimators.
    • Expose Head as public API.
    • Fixes critical bugs that help with DenseFeatures usability in TF2
  • tf.data :
    • Promoting unbatch from experimental to core API.
    • Adding support for datasets as inputs to from_tensors and from_tensor_slices and batching and unbatching of nested datasets.
  • tf.keras :
    • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format, which allows the saved checkpoints to be compatible with model.load_weights .
    • Saving a Keras Model using tf.saved_model.save now saves the list of variables, trainable variables, regularization losses, and the call function.
    • Deprecated tf.keras.experimental.export_saved_model and tf.keras.experimental.function . Please use tf.keras.models.save_model(. save_format=’tf’) and tf.keras.models.load_model instead.
    • Add an implementation=3 mode for tf.keras.layers.LocallyConnected2D and tf.keras.layers.LocallyConnected1D layers using tf.SparseTensor to store weights, allowing a dramatic speedup for large sparse models.
    • Enable the Keras compile API experimental_run_tf_function flag by default. This flag enables single training/eval/predict execution path. With this 1. All input types are converted to Dataset . 2. When distribution strategy is not specified this goes through the no-op distribution strategy path. 3. Execution is wrapped in tf.function unless run_eagerly=True is set in compile.
    • Raise error if batch_size argument is used when input is dataset/generator/keras sequence.
  • tf.lite
    • Add GATHER support to NN API delegate.
    • tflite object detection script has a debug mode.
    • Add delegate support for QUANTIZE .
    • Added evaluation script for COCO minival.
    • Add delegate support for QUANTIZED_16BIT_LSTM .
    • Converts hardswish subgraphs into atomic ops.
  • Add support for defaulting the value of cycle_length argument of tf.data.Dataset.interleave to the number of schedulable CPU cores.
  • parallel_for : Add converter for MatrixDiag .
  • Add narrow_range attribute to QuantizeAndDequantizeV2 and V3.
  • Added new op: tf.strings.unsorted_segment_join .
  • Add HW acceleration support for topK_v2 .
  • Add new TypeSpec classes.
  • CloudBigtable version updated to v0.10.0.
  • Expose Head as public API.
  • Update docstring for gather to properly describe the non-empty batch_dims case.
  • Added tf.sparse.from_dense utility function.
  • Improved ragged tensor support in TensorFlowTestCase .
  • Makes the a-normal form transformation in Pyct configurable as to which nodes are converted to variables and which are not.
  • ResizeInputTensor now works for all delegates.
  • Add EXPAND_DIMS support to NN API delegate TEST: expand_dims_test
  • tf.cond emits a StatelessIf op if the branch functions are stateless and do not touch any resources.
  • tf.cond , tf.while and if and while in AutoGraph now accept a nonscalar predicate if has a single element. This does not affect non-V2 control flow.
  • tf.while_loop emits a StatelessWhile op if the cond and body functions are stateless and do not touch any resources.
  • Refactors code in Quant8 LSTM support to reduce TFLite binary size.
  • Add support of local soft device placement for eager op.
  • Add HW acceleration support for LogSoftMax .
  • Added a function nested_value_rowids for ragged tensors.
  • Add guard to avo >tf.math.cumulative_logsumexp operation .
  • Add tf.ragged.stack .
  • Fix memory allocation problem when calling AddNewInputConstantTensor .
  • Delegate application failure leaves interpreter in valid state.
  • Add check for correct memory alignment to MemoryAllocation::MemoryAllocation() .
  • Extracts NNAPIDelegateKernel from nnapi_delegate.cc
  • Added support for FusedBatchNormV3 in converter.
  • A ragged to dense op for directly calculating tensors.
  • Fix accidental quadratic graph construction cost in graph-mode tf.gradients() .
  • The precision_mode argument to TrtGraphConverter is now case insensitive.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

a6802739, Aaron Ma, Abdullah Selek, Abolfazl Shahbazi, Ag Ramesh, Albert Z. Guo, Albin Joy, Alex Itkes, Alex Sergeev, Alexander Pivovarov, Alexey Romanov, alhkad, Amit Srivastava, amoitra, Andrew Lihonosov, Andrii Prymostka, Anuj Rawat, Astropeak, Ayush Agrawal, Bairen Yi, Bas Aarts, Bastian Eichenberger, Ben Barsdell, Benjamin Peterson, bhack, Bharat Raghunathan, Bhavani Subramanian, Bryan Cutler, candy.dc, Cao Zongyan, Captain-Pool, Casper Da Costa-Luis, Chen Guoyin, Cheng Chang, chengchingwen, Chong Yan, Choong Yin Thong, Christopher Yeh, Clayne Robison, Coady, Patrick, Dan Ganea, David Norman, Denis Khalikov, Deven Desai, Diego Caballero, Duncan Dean, Duncan Riach, Dwight J Lyle, Eamon Ito-Fisher, eashtian3, EFanZh, ejot, Elroy Ashtian Jr, Eric Schweitz, Fangjun Kuang, Fei Hu, fo40225, formath, Fred Reiss, Frederic Bastien, Fredrik Knutsson, G. Hussain Chinoy, Gabriel, gehring, George Grzegorz Pawelczak, Gianluca Varisco, Gleb Popov, Greg Peatfield, Guillaume Klein, Gurpreet Singh, Gustavo Lima Chaves, haison, Haraldur TóMas HallgríMsson, HarikrishnanBalagopal, HåKon Sandsmark, I-Hong, Ilham Firdausi Putra, Imran Salam, Jason Zaman, Jason Zavaglia, jayhpark530, jefby, Jeff Daily, Jeffrey Poznanovic, Jekyll Lai, Jeroen BéDorf, Jerry Shih, jerryyin, jiakai, JiangXIAO, Joe Bowser, Joel Shapiro, Johan Gunnarsson, Jojimon Varghese, Joon, Josh Beal, Julian Niedermeier, Jun Wan, Junqin Zhang, Junyuan Xie, Justin Tunis, Kaixi Hou, Karl Lessard, Karthik Muthuraman, Kbhute-Ibm, khanhlvg, Koock Yoon, kstuedem, Kyuwon Kim, Lakshay Tokas, leike666666, leonard951, Leslie-Fang, Leslie-Fang-Intel, Li, Guizi, Lukas Folle, Lukas Geiger, Mahmoud Abuzaina, Manraj Singh Grover, Margaret Maynard-Reid, Mark Ryan, Matt Conley, Matthew Bentham, Matthew Denton, mbhuiyan, mdfaijul, Mei Jie, merturl, MichaelKonobeev, Michal W. Tarnowski, minds, mpppk, musikisomorphie, Nagy Mostafa, Nayana Thorat, Neil, Niels Ole Salscheider, Niklas SilfverströM, Niranjan Hasabnis, ocjosen, olramde, Pariksheet Pinjari, Patrick J. Lopresti, Patrik Gustavsson, per1234, PeterLee, Phan Van Nguyen Duc, Phillip Kravtsov, Pooya Davoodi, Pranav Marathe, Putra Manggala, Qingqing Cao, Rajeshwar Reddy T, Ramon ViñAs, Rasmus Diederichsen, Reuben Morais, richardbrks, robert, RonLek, Ryan Jiang, saishruthi, Saket Khandelwal, Saleem Abdulrasool, Sami Kama, Sana-Damani, Sergii Khomenko, Severen Redwood, Shubham Goyal, Sigrid Keydana, Siju Samuel, sleighsoft, smilu97, Son Tran, Srini511, srinivasan.narayanamoorthy, Sumesh Udayakumaran, Sungmann Cho, Tae-Hwan Jung, Taehoon Lee, Takeshi Watanabe, TengLu, terryky, TheMindVirus, ThisIsIsaac, Till Hoffmann, Timothy Liu, Tomer Gafner, Tongxuan Liu, Trent Lo, Trevor Morris, Uday Bondhugula, Vasileios Lioutas, vbvg2008, Vishnuvardhan Janapati, Vivek Suryamurthy, Wei Wang, Wen-Heng (Jack) Chung, wenxizhu, William D. Irons, winstonq, wyzhao, Xiaoming (Jason) Cui, Xinan Jiang, Xinping Wang, Yann-Yy, Yasir Modak, Yong Tang, Yongfeng Gu, Yuchen Ying, Yuxin Wu, zyeric, 王振华 (Zhenhua Wang)

goldiegadde released this Sep 11, 2020 · 86 commits to r2.0 since this release


Release 2.0.0-rc1

Major Features and Improvements

TensorFlow 2.0 focuses on simplicity and ease of use, featuring updates like:

  • Easy model building with Keras and eager execution.
  • Robust model deployment in production on any platform.
  • Powerful experimentation for research.
  • API simplification by reducing duplication and removing deprecated endpoints.

For details on best practices with 2.0, see the Effective 2.0 guide

For information on upgrading your existing TensorFlow 1.x models, please refer to our Upgrade and Migration guides. We have also released a collection of tutorials and getting started guides.

Highlights

  • TF 2.0 delivers Keras as the central high level API used to build and train models. Keras provides several model-building APIs such as Sequential, Functional, and Subclassing along with eager execution, for immediate iteration and intuitive debugging, and tf.data, for building scalable input pipelines. Checkout guide for additional details.
  • Distribution Strategy: TF 2.0 users will be able to use the tf.distribute.Strategy API to distribute training with minimal code changes, yielding great out-of-the-box performance. It supports distributed training with Keras model.fit, as well as with custom training loops. Multi-GPU support is available, along with experimental support for multi worker and Cloud TPUs. Check out the guide for more details.
  • Functions, not Sessions. The traditional declarative programming model of building a graph and executing it via a tf.Session is discouraged, and replaced by writing regular Python functions. Using the tf.function decorator, such functions can be turned into graphs which can be executed remotely, serialized, and optimized for performance.
  • Unification of tf.train.Optimizers and tf.keras.Optimizers. Use tf.keras.Optimizers for TF2.0. compute_gradients is removed as public API, and use GradientTape to compute gradients.
  • AutoGraph translates Python control flow into TensorFlow expressions, allowing users to write regular Python inside tf.function -decorated functions. AutoGraph is also applied in functions used with tf.data, tf.distribute and tf.keras APIs.
  • Unification of exchange formats to SavedModel. All TensorFlow ecosystem projects (TensorFlow Lite, TensorFlow JS, TensorFlow Serving, TensorFlow Hub) accept SavedModels. Model state should be saved to and restored from SavedModels.
  • API Changes: Many API symbols have been renamed or removed, and argument names have changed. Many of these changes are motivated by consistency and clarity. The 1.x API remains available in the compat.v1 module. A list of all symbol changes can be found here.
  • API clean-up, included removing tf.app , tf.flags , and tf.logging in favor of absl-py.
  • No more global variables with helper methods like tf.global_variables_initializer and tf.get_global_step .
  • Add toggles tf.enable_control_flow_v2() and tf.disable_control_flow_v2() for enabling/disabling v2 control flow.
  • Enable v2 control flow as part of tf.enable_v2_behavior() and TF2_BEHAVIOR=1 .
  • Fixes autocomplete for most TensorFlow API references by switching to use relative imports in API __init__.py files.

Breaking Changes

Many backwards incompatible API changes have been made to clean up the APIs and make them more consistent.

  • TensorFlow 2.0.0 is built using devtoolset7 (GCC7) on Ubuntu 16. This may lead to ABI incompatibilities with extensions built against earlier versions of TensorFlow.
  • Tensorflow code now produces 2 different pip packages: tensorflow_core containing all the code (in the future it will contain only the private implementation) and tensorflow which is a virtual pip package doing forwarding to tensorflow_core (and in the future will contain only the public API of tensorflow). We don’t expect this to be breaking, unless you were importing directly from the implementation.
  • tf.contrib has been deprecated, and functionality has been either migrated to the core TensorFlow API, to an ecosystem project such as tensorflow/addons or tensorflow/io, or removed entirely.
  • Remove tf.contrib.timeseries dependency on TF distributions.
  • Replace contrib references with tf.estimator.experimental.* for apis in early_stopping.py.
  • Premade estimators in the tf.estimator.DNN/Linear/DNNLinearCombined family have been updated to use tf.keras.optimizers instead of the tf.compat.v1.train.Optimizer s. If you do not pass in an optimizer= arg or if you use a string, the premade estimator will use the Keras optimizer. This is checkpoint breaking, as the optimizers have separate variables. A checkpoint converter tool for converting optimizers is included with the release, but if you want to avoid any change, switch to the v1 version of the estimator: tf.compat.v1.estimator.DNN/Linear/DNNLinearCombined* .
  • Default aggregation for canned Estimators is now SUM_OVER_BATCH_SIZE . To maintain previous default behavior, please pass SUM as the loss aggregation method.
  • Canned Estimators don’t support input_layer_partitioner arg in the API. If you have this arg, you will have to switch to tf.compat.v1 canned Estimators .
  • Estimator.export_savedmodel has been renamed export_saved_model .
  • When saving to SavedModel, Estimators will strip default op attributes. This is almost always the correct behavior, as it is more forwards compatible, but if you require that default attributes are saved with the model, please use tf.compat.v1.Estimator .
  • Feature Columns have been upgraded to be more Eager-friendly and to work with Keras. As a result, tf.feature_column.input_layer has been deprecated in favor of tf.keras.layers.DenseFeatures . v1 feature columns have direct analogues in v2 except for shared_embedding_columns , which are not cross-compatible with v1 and v2. Use tf.feature_column.shared_embeddings instead.
  • OMP_NUM_THREADS is no longer used by the default Keras config. To configure the number of threads, use tf.config.threading APIs.
  • tf.keras.model.save_model and model.save now defaults to saving a TensorFlow SavedModel. HDF5 files are still supported.
  • Deprecated tf.keras.experimental.export_saved_model and tf.keras.experimental.function . Please use tf.keras.models.save_model(. save_format=’tf’) and tf.keras.models.load_model instead.
  • Layers now default to float32, and automatically cast their inputs to the layer’s dtype. If you had a model that used float64, it will probably silently use float32 in TensorFlow 2, and a warning will be issued that starts with «Layer is casting an input tensor from dtype float64 to the layer’s dtype of float32». To fix, either set the default dtype to float64 with tf.keras.backend.set_floatx(‘float64′) , or pass dtype=’float64’ to each of the Layer constructors. See tf.keras.layers.Layer for more information.
  • Removed lite.OpHint , lite.experimental , and lite.constant from 2.0 API.

Tensors are no longer hashable, but instead compare element-wise with == and != . Use tf.compat.v1.disable_tensor_equality() to return to the previous behavior.

Performing equality operations on Tensors or Variables with incompatible shapes an exception is no longer thrown. Instead __eq__ returns False and __ne__ returns True.

Removed tf.string_split from v2 API.

Deprecated the use of constraint= and .constraint with ResourceVariable.

Add UnifiedGRU as the new GRU implementation for tf2.0. Change the default recurrent activation function for GRU from hard_sigmoid to sigmoid , and reset_after to True in 2.0. Historically recurrent activation is hard_sigmoid since it is fast than ‘sigmo , reset_after=False) to fallback to 1.x behavior.

CUDNN_INSTALL_PATH , TENSORRT_INSTALL_PATH , NCCL_INSTALL_PATH , NCCL_HDR_PATH are deprecated. Use TF_CUDA_PATHS instead which supports a comma-separated list of base paths that are searched to find CUDA libraries and headers.

Refer to our public project status tracker and issues tagged with 2.0 on GitHub for insight into recent issues and development progress.

If you experience any snags when using TF 2.0, please let us know at the TF 2.0 Testing User Group. We have a support mailing list as well as weekly testing meetings, and would love to hear your migration feedback and questions.

Bug Fixes and Other Changes

  • Expose tf.contrib.proto.* ops in tf.io (they will exist in TF2)
  • Add support for TensorArrays to tf.data Dataset .
  • Integrate Ragged Tensors with tf.data .
  • All core and experimental tf.data transformations that input user-defined functions can span multiple devices now.
  • Extending the TF 2.0 support for shuffle(. reshuffle_each_iteration=True) and cache() to work across different Python iterators for the same dataset.
  • Removing the experimental_numa_aware option from tf.data.Options .
  • Add num_parallel_reads and passing in a Dataset containing filenames into TextLineDataset and FixedLengthRecordDataset .
  • Add support for defaulting the value of cycle_length argument of tf.data.Dataset.interleave to the number of schedulable CPU cores.
  • Promoting tf.data.experimental.enumerate_dataset to core as tf.data.Dataset.enumerate .
  • Promoting tf.data.experimental.unbatch to core as tf.data.Dataset.unbatch .
  • Adds option for introducing slack in the pipeline to reduce CPU contention, via tf.data.Options().experimental_slack = True
  • Added experimental support for parallel batching to batch() and padded_batch() . This functionality can be enabled through tf.data.Options()
  • Support cancellation of long-running reduce .
  • Now we use dataset node name as prefix instead of the op name, to identify the component correctly in metrics, for pipelines with repeated components.
  • Improve the performance of datasets using from_tensors() .
  • Promoting unbatch from experimental to core API.
  • Adding support for datasets as inputs to from_tensors and from_tensor_slices and batching and unbatching of nested datasets.
  • Enable tf.distribute.experimental.MultiWorkerMirroredStrategy working in eager mode.
  • Callbacks are supported in MultiWorkerMirroredStrategy .
  • Disable run_eagerly and distribution strategy if there are symbolic tensors added to the model using add_metric or add_loss .
  • Loss and gradients should now more reliably be correctly scaled w.r.t. the global batch size when using a tf.distribute.Strategy .
  • Set default loss reduction as AUTO for improving reliability of loss scaling with distribution strategy and custom training loops. AUTO indicates that the reduction option will be determined by the usage context. For almost all cases this defaults to SUM_OVER_BATCH_SIZE . When used in distribution strategy scope, outside of built-in training loops such as tf.keras compile and fit , we expect reduction value to be ‘None’ or ‘SUM’. Using other values will raise an error.
  • Support for multi-host ncclAllReduce in Distribution Strategy.
  • Replace tf.contrib.estimator.add_metrics with tf.estimator.add_metrics
  • Use tf.compat.v1.estimator.inputs instead of tf.estimator.inputs
  • Replace contrib references with tf.estimator.experimental.* for apis in early_s in Estimator
  • Canned Estimators will now use keras optimizers by default. An error will be raised if tf.train.Optimizers are used, and you will have to switch to tf.keras.optimizers or tf.compat.v1 canned Estimators.
  • A checkpoint converter for canned Estimators has been provided to transition canned Estimators that are warm started from tf.train.Optimizers to tf.keras.optimizers .
  • Losses are scaled in canned estimator v2 and not in the optimizers anymore. If you are using Estimator + distribution strategy + optimikzer v1 then the behavior does not change. This implies that if you are using custom estimator with optimizer v2, you have to scale losses. We have new utilities to help scale losses tf.nn.compute_average_loss , tf.nn.scale_regularization_loss .
  • Premade models (including Linear and WideDeep) have been introduced for the purpose of replacing Premade estimators.
  • Model saving changes
  • model.save and tf.saved_model.save may now save to the TensorFlow SavedModel format. The model can be restored using tf.keras.models.load_model . HDF5 files are still supported, and may be used by specifying save_format=»h5″ when saving.
  • Raw TensorFlow functions can now be used in conjunction with the Keras Functional API during model creation. This obviates the need for users to create Lambda layers in most cases when using the Functional API. Like Lambda layers, TensorFlow functions that result in Variable creation or assign ops are not supported.
  • Add support for passing list of lists to the metrics argument in Keras compile .
  • Add tf.keras.layers.AbstractRNNCell as the preferred implementation for RNN cells in TF v2. User can use it to implement RNN cells with custom behavior.
  • Keras training and validation curves are shown on the same plot when using the TensorBoard callback.
  • Switched Keras fit/evaluate/predict execution to use only a single unified path by default unless eager execution has been explicitly disabled, regardless of input type. This unified path places an eager-friendly training step inside of a tf.function . With this
  1. All input types are converted to Dataset .
  2. The path assumes there is always a distribution strategy. when distribution strategy is not specified the path uses a no-op distribution strategy.
  3. The training step is wrapped in tf.function unless run_eagerly=True is set in compile. The single path execution code does not yet support all use cases. We fallback to the existing v1 execution paths if your model contains the following:
  4. sample_weight_mode in compile
  5. weighted_metrics in compile
  6. v1 optimizer
  7. target tensors in compile
    If you are experiencing any issues because of this change, please inform us (file an issue) about your use case and you can unblock yourself by setting experimental_run_tf_function=False in compile meanwhile. We have seen couple of use cases where the model usage pattern is not as expected and would not work with this change.
  8. output tensors of one layer is used in the constructor of another.
  9. symbolic tensors outside the scope of the model are used in custom loss functions.
    The flag can be disabled for these cases and ideally the usage pattern will need to be fixed.
  • Mark Keras set_session as compat.v1 only.
  • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format , which allows the saved checkpoints to be compatible with model.load_weights .
  • keras.backend.resize_images (and consequently, keras.layers.Upsampling2D ) behavior has changed, a bug in the resizing implementation was fixed.
  • Add an implementation=3 mode for tf.keras.layers.LocallyConnected2D and tf.keras.layers.LocallyConnected1D layers using tf.SparseTensor to store weights, allowing a dramatic speedup for large sparse models.
  • Raise error if batch_size argument is used when input is dataset/generator/keras sequence.
  • Update TF 2.0 keras.backend.name_scope to use TF 2.0 name_scope .
  • Add v2 module aliases for losses, metrics, initializers and optimizers: tf.losses = tf.keras.losses & tf.metrics = tf.keras.metrics & tf.initializers = tf.keras.initializers & tf.optimizers = tf.keras.optimizers .
  • Updates binary cross entropy logic in Keras when input is probabilities. Instead of converting probabilities to logits, we are using the cross entropy formula for probabilities.
  • Added public APIs for cumsum and cumprod keras backend functions.
  • Add support for temporal sample weight mode in subclassed models.
  • Raise ValueError if an integer is passed to the training APIs.
  • Added fault-tolerance support for training Keras model via model.fit() with MultiWorkerMirroredStrategy , tutorial available.
  • Custom Callback tutorial is now available.
  • To train with tf.distribute , Keras api is recommended over estimator.
  • steps_per_epoch and steps arguments are supported with numpy arrays.
  • New error message when unexpected keys are used in sample_weight/class_weight dictionaries
  • Losses are scaled in Keras compile/fit and not in the optimizers anymore. If you are using custom training loop, we have new utilities to help scale losses tf.nn.compute_average_loss , tf.nn.scale_regularization_loss .
  • Layer apply and add_variable APIs are deprecated.
  • Added support for channels first data format in cross entropy losses with logits and support for tensors with unknown ranks.
  • Error messages will be raised if add_update , add_metric , add_loss , activity regularizers are used inside of a control flow branch.
  • New loss reduction types:
    1. AUTO : Indicates that the reduction option will be determined by the usage context. For almost all cases this defaults to SUM_OVER_BATCH_SIZE . When used with tf.distribute.Strategy , outside of built-in training loops such as tf.keras compile and fit , we expect reduction value to be SUM or NONE . Using AUTO in that case will raise an error.
    2. NONE : Weighted losses with one dimension reduced (axis=-1, or axis specified by loss function). When this reduction type used with built-in Keras training loops like fit / evaluate , the unreduced vector loss is passed to the optimizer but the reported loss will be a scalar value.
    3. SUM : Scalar sum of weighted losses. 4. SUM_OVER_BATCH_SIZE : Scalar SUM divided by number of elements in losses. This reduction type is not supported when used with tf.distribute.Strategy outside of built-in training loops like tf.keras compile / fit .
  • Wraps losses passed to the compile API (strings and v1 losses) which are not instances of v2 Loss class in LossWrapper class. => All losses will now use SUM_OVER_BATCH_SIZE reduction as default.
  • model.add_loss(symbolic_tensor) should work in ambient eager.
  • Update metric name to always reflect what the user has given in compile. Affects following cases
    1. When name is given as ‘accuracy’/’crossentropy’
    2. When an aliased function name is used eg. ‘mse’
    3. Removing the weighted prefix from weighted metric names.
  • Allow non-Tensors through v2 losses.
  • Add v2 sparse categorical crossentropy metric.
  • Add v2 APIs for AUCCurve and AUCSummationMethod enums.
  • add_update can now be passed a zero-arg callable in order to support turning off the update when setting trainable=False on a Layer of a Model compiled with run_eagerly=True .
  • Standardize the LayerNormalization API by replacing the args norm_axis and params_axis with axis .
  • Fixed critical bugs that help with DenseFeatures usability in TF2
  • Added evaluation script for COCO minival
  • Add delegate support for QUANTIZE .
  • Add GATHER support to NN API delegate.
  • Added support for TFLiteConverter Python API in 2.0. Contains functions from_saved_model, from_keras_file, and from_concrete_functions.
  • Add EXPAND_DIMS support to NN API delegate TEST.
  • Add narrow_range attribute to QuantizeAndDequantizeV2 and V3.
  • Added support for tflite_convert command line tool in 2.0.
  • Post-training quantization tool supports quantizing weights shared by multiple operations. The models made with versions of this tool will use INT8 types for weights and will only be executable interpreters from this version onwards.
  • Post-training quantization tool supports fp16 weights and GPU delegate acceleration for fp16.
  • Add delegate support for QUANTIZED_16BIT_LSTM .
  • Extracts NNAPIDelegateKernel from nnapi_delegate.cc
  • Fix accidental quadratic graph construction cost in graph-mode tf.gradients() .
  • ResourceVariable’s gather op supports batch dimensions.
  • ResourceVariable support for gather_nd .
  • ResourceVariable and Variable no longer accepts constraint in the constructor, nor expose it as a @Property.
  • Added gradient for SparseToDense op.
  • Expose a flag that allows the number of threads to vary across Python benchmarks.
  • image.resize in 2.0 now supports gradients for the new resize kernels.
  • image.resize now considers proper pixel centers and has new kernels (incl. anti-aliasing).
  • Renamed tf.image functions to remove duplicate «image» where it is redundant.
  • Variadic reduce is supported on CPU Variadic reduce is supported on CPU
  • Remove unused StringViewVariantWrapper .
  • Delete unused Fingerprint64Map op registration
  • Add broadcasting support to tf.matmul .
  • Add C++ Gradient for BatchMatMulV2 .
  • Add tf.math.cumulative_logsumexp operation.
  • Add ellipsis (. ) support for tf.einsum() .
  • Add expand_composites argument to all nest.* methods.
  • Added strings.byte_split .
  • Add a new «result_type» parameter to tf.strings.split .
  • Add name argument to tf.string_split and tf.strings_split .
  • Extend tf.strings.split to support inputs with any rank.
  • Added tf.random.binomial .
  • Added key and skip methods to random.experimental.Generator .
  • Extend tf.function with basic support for CompositeTensors arguments (such as SparseTensor and RaggedTensor ).
  • parallel_for.pfor : add converters for Softmax, LogSoftmax, IsNaN, All, Any, and MatrixSetDiag.
  • parallel_for : add converters for LowerTriangularSolve and Cholesky.
  • parallel_for : add converters for LogMatrixDeterminant and MatrixBandPart .
  • parallel_for : Add converter for MatrixDiag .
  • parallel_for : Add converters for OneHot , LowerBound , UpperBound .
  • parallel_for : add converter for BroadcastTo .
  • Add pfor converter for Squeeze .
  • Add RaggedTensor.placeholder() .
  • Add ragged tensor support to tf.squeeze .
  • Update RaggedTensors to support int32 row_splits.
  • Allow LinearOperator.solve to take a LinearOperator .
  • Allow all dtypes for LinearOperatorCirculant .
  • Introduce MaxParallelism method
  • Add LinearOperatorHouseholder .
  • Adds Philox support to new stateful RNG’s XLA path.
  • Add TensorSpec support for CompositeTensors.
  • Add tf.linalg.tridiagonal_solve op.
  • Added partial_pivoting input parameter to tf.linalg.tridiagonal_solve .
  • Added gradient to tf.linalg.tridiagonal_solve .
  • Add tf.linalg.tridiagonal_mul op .
  • Added GPU implementation of tf.linalg.tridiagonal_matmul .
  • Add LinearOperatorToeplitz .
  • Upgraded LIBXSMM to version 1.11.
  • Uniform processing of quantized embeddings by Gather and EmbeddingLookup Ops
  • Correct a misstatement in the documentation of the sparse softmax cross entropy logit parameter.
  • Add tf.ragged.boolean_mask .
  • tf.switch_case added, which selects a branch_fn based on a branch_index.
  • The C++ kernel of gather op supports batch dimensions.
  • Fixed default value and documentation for trainable arg of tf.Variable.
  • EagerTensor now supports numpy buffer interface for tensors.
  • This change bumps the version number of the FullyConnected Op to 5.
  • Added new op: tf.strings.unsorted_segment_join .
  • Add HW acceleration support for topK_v2 .
  • CloudBigtable version updated to v0.10.0 BEGIN_PUBLIC CloudBigtable version updated to v0.10.0.
  • Expose Head as public API.
  • Added tf.sparse.from_dense utility function.
  • Improved ragged tensor support in TensorFlowTestCase .
  • Added a function nested_value_rowids for ragged tensors.
  • Add tf.ragged.stack .
  • Makes the a-normal form transformation in Pyct configurable as to which nodes are converted to variables and which are not.
  • ResizeInputTensor now works for all delegates.
  • tf.cond emits a StatelessIf op if the branch functions are stateless and do not touch any resources.
  • Add support of local soft device placement for eager op.
  • Pass partial_pivoting to the _TridiagonalSolveGrad .
  • Add HW acceleration support for LogSoftMax .
  • Add guard to avo >AddNewInputConstantTensor .
  • Delegate application failure leaves interpreter in valid state
  • tf.while_loop emits a StatelessWhile op if the cond and body functions are stateless and do not touch any resources.
  • tf.cond, tf.while and if and while in AutoGraph now accept a nonscalar predicate if has a single element. This does not affec non-V2 control flow.
  • Fix potential security vulnerability where decoding variant tensors from proto could result in heap out of bounds memory access.
  • Only create a GCS directory object if the object does not already exist.
  • Introduce dynamic constructor argument in Layer and Model, which should be set to True when using imperative control flow in the call method.
  • Begin adding Go wrapper for C Eager API.
  • XLA HLO graphs can be inspected with interactive_graphviz tool now.
  • Add dataset ops to the graph (or create kernels in Eager execution) during the python Dataset object creation instead doing it during Iterator creation time.
  • Add batch_dims argument to tf.gather .
  • The behavior of tf.gather is now correct when axis=None and batch_dims batch_dims case.
  • Removing of dtype in the constructor of initializers and partition_info in call.
  • Add tf.math.nextafter op.
  • Turn on MKL-DNN contraction kernels by default. MKL-DNN dynamically dispatches the best kernel implementation based on CPU vector architecture. To disable them, build with —define=tensorflow_mkldnn_contraction_kernel=0 .
  • tf.linspace(start, stop, num) now always uses «stop» as last value (for num > 1)
  • Added top-k to precision and recall to keras metrics.
  • Add a ragged size op and register it to the op dispatcher
  • Transitive dependencies on : pooling_ops were removed. Some users may need to add explicit dependencies on : pooling_ops if they reference the operators from that library.
  • Add CompositeTensor base class.
  • Malformed gif images could result in an access out of bounds in the color palette of the frame. This has been fixed now
  • Add templates and interfaces for creating lookup tables
  • Tensor::UnsafeCopyFromInternal deprecated in favor Tensor::BitcastFrom .
  • In map_vectorization optimization, reduce the degree of parallelism in the vectorized map node.
  • Add variant wrapper for absl::string_view .
  • Add OpKernels for some stateless maps.
  • DType is no longer convertible to an int. Use dtype.as_datatype_enum instead of int(dtype) to get the same result.
  • Support both binary and -1/1 label input in v2 hinge and squared hinge losses.
  • Added LinearOperator.adjoint and LinearOperator.H (alias).
  • Expose CriticalSection in core as tf.CriticalSection .
  • Enhanced graphviz output.
  • Add opkernel templates for common table operations.
  • Fix callbacks do not log values in eager mode when a deferred build model is used.
  • SignatureDef util functions have been deprecated.
  • Update Fingerprint64Map to use aliases
  • Add legacy string flat hash map op kernels.
  • Add support for add_metric in the graph function mode.
  • Updating cosine similarity loss — removed the negate sign from cosine similarity.
  • Changed default for gradient accumulation for TPU embeddings to true.
  • Adds summary trace API for collecting graph and profile information.
Мастер Йода рекомендует:  Безопасности тоже бывает много в Интернет

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

1e100, a6802739, Abolfazl Shahbazi, Adam Weiss, Ag Ramesh, Alan Du, Albin Joy, Amit, Amit Srivastava, Andy Craze, Anshuman Tripathy, Armen Poghosov, armenpoghosov, Arpit Shah, Ashwin Ramaswami, Aurelien Geron, AuréLien Geron, aweers, awesomealex1, Bairen Yi, Ben Barsdell, Bhavani Subramanian, Brandon Carter, candy.dc, Chao Liu, Clayne Robison, csukuangfj, Dan Jarvis, Dan Lazewatsky, Daniel Ingram, Dave Airlie, David Norman, Dayananda V, Denis Khalikov, Deven Desai, Dheeraj Rajaram Reddy, dmitrievanthony, Drew Szurko, Duncan Riach, Fei Hu, Felix Lemke, Filip Matzner, fo40225, frreiss, Gautam, gehring, Grzegorz George Pawelczak, Grzegorz Pawelczak, HanGuo97, Hari Shankar, hehongliang, Heungsub Lee, Hoeseong Kim, I-Hong Jhuo, Ilango R, Innovimax, Jacky Ko, Jakub Lipinski, jcf94, Jeff Poznanovic, Jia Qingtong, Jiankang, Joe Q, Joe Quadrino, Jonas Rauber, Jonathan Kyl, Joppe Geluykens, Joseph Friedman, jtressle, jwu, K. Hodges, kaixih, Karl Lessard, Karl Weinmeister, Kashif Rasul, kjopek, Koan-Sin Tan, kouml, ktaebum, Laurent Le Brun, Li, Guizi, Loo Rong Jie, Lucas Hendren, Lukas Geiger, Luke Han, Mahmoud Abuzaina, manhyuk, Marco Gaido, Marek Drozdowski, Mark Ryan, mars20, Mateusz Chudyk, Matt Conley, MattConley, mbhuiyan, mdfaijul, Melissa Grueter, Michael KäUfl, MickaëL Schoentgen, Miguel Morin, Mike Arpaia, nammbash, Natalia Gimelshein, Nayana-Ibm, neargye, Nehal J Wani, Niels Ole Salscheider, Niranjan Hasabnis, Nutti, olicht, P Sudeepam, Paige Bailey, Palmer Lao, Pariksheet Pinjari, Pavel Samolysov, Pooya Davoodi, Ryan Jiang, Samantha Andow, Sami Kama, Saurabh Deoras, Shahzad Lone, Shashi, Siju, Siju Samuel, Snease-Abq, Spencer Schaber, srinivasan.narayanamoorthy, Steve Lang, Steve Nesae, Supriya Rao, Taylor Jakobson, Taylor Thornton, ThisIsPIRI, Thomas Deegan, tomguluson92, Tongxuan Liu, Vagif, vcarpani, Vikram Tiwari, Vishwak Srinivasan, Vitor-Alves, wangsiyu, WeberXie, WeijieSun, Wen-Heng (Jack) Chung, wenxizhu, William D. Irons, Yan Facai (颜发才), ymodak, Yong Tang, Younes Khoudli, Yuan Lin, Yves-Noel Weweler, zjjott, 卜居, 王振华 (Wang Zhenhua),

4d55397500, a6802739, Abdullah Selek, abenmao, Adam Richter, Ag Ramesh, Albin Joy, Alex, Alex Itkes, Alex Sergeev, Alexander Pivovarov, Alexey Romanov, alhkad, Aman Patel, Amit, Amit Kumar Jaiswal, Amit Srivastava, amoitra, Andreas Eberle, Andrew Lihonosov, Anthony Hsu, Anthony Platanios, Anuj Rawat, arp95, Arpit Shah, Astropeak, Augustina Ragwitz, Aurelien Geron, AuréLien Geron, avasid, aweers, Ayush Agrawal, Bas Aarts, Bastian Eichenberger, Bayberry Z, Ben Barsdell, Benjamin Peterson, bhack, Bharat Raghunathan, Bhavani Subramanian, Bin Fan, blairhan, BléNesi Attila, Bodin-E, Bryan Cutler, Cao Zongyan, Casper Da Costa-Luis, Chen Guoyin, chenchc, chengchingwen, chie8842, Christian Hansen, Christoph Boeddeker, Christopher Yeh, Clayne Robison, Coady, Patrick, crafet, ctiijima, Daniel Rasmussen, Daniel Salvadori, David Norman, delock, Denis Khalikov, Deven Desai, Diego Caballero, Donovan Ong, Duncan Dean, Duncan Riach, Dustin Neighly, Dwight J Lyle, Eamon Ito-Fisher, eashtian3, Edward Forgacs, EFanZh, ejot, Elroy Ashtian Jr, Eric Schweitz, Evgeniy Polyakov, Fangjun Kuang, Federico Martinez, Fei Hu, Filip Matzner, FlashTek, fo40225, formath, FrançOis Chollet, Fred Reiss, Frederic Bastien, Fredrik Knutsson, G. Hussain Chinoy, Gabriel, gehring, Geoffrey Irving, George Grzegorz Pawelczak, George Sterpu, Gianluca Varisco, Gleb Popov, Greg Peatfield, Guillaume Klein, Gurpreet Singh, Gustavo Lima Chaves, Gyoung-Yoon Ryoo, haison, Hanton Yang, Haraldur TóMas HallgríMsson, Huan Li (李卓桓), HåKon Sandsmark, I-Hong, Ilham Firdausi Putra, Imran Salam, Irene Dea, Ivan Habernal, Jacky, Jason Zaman, Jason Zavaglia, jayhpark530, jefby, Jeff Daily, Jeffrey Poznanovic, Jekyll Lai, jer, Jeroen BéDorf, jerryyin, jhalakp, jiakai, JiangXIAO, Joe Bowser, Joel Shapiro, Johan Gunnarsson, Jojimon Varghese, Jonathan, Joon, Josh Beal, Julian Niedermeier, Junqin Zhang, Justin Dujardin, Justin Tunis, Kaixi Hou, Karthik Muthuraman, Kay Zhu, Kbhute-Ibm, KDR, Keno Fischer, Kevin Mader, khanhlvg, Kilaru Yasaswi Sri Chandra Gandhi, Koock Yoon, Kyuwon Kim, Lakshay Tokas, leike666666, leonard951, Leslie-Fang, Letian Kang, Li, Guizi, Lukas Folle, Lukas Geiger, luxupu, lvli, Ma, Guokai, Mahmoud Abuzaina, Maksym Kysylov, Mandar Deshpande, Manraj Singh Grover, Margaret Maynard-Reid, Mark Ryan, Matt Conley, mbhuiyan, mdfaijul, Mei Jie, merturl, MichaelKonobeev, Michal W. Tarnowski, Mihail Salnikov, Mikalai Drabovich, Mike Holcomb, minds, monklof, Moses Marin, mpppk, Mr. Metal, Mshr-H, musikisomorphie, nammbash, Nathan Luehr, Nayana Thorat, Neeraj Pradhan, Neil, Nick, Nick Lewycky, Niels Ole Salscheider, Niklas SilfverströM, Niranjan Hasabnis, Nuka-137, ocjosen, omeir1, P Sudeepam, Pan Daoxin, Pariksheet Pinjari, Pasquale Minervini, Patrick J. Lopresti, Patrik Gustavsson, Pavel Akhtyamov, PENGWA, per1234, PeterLee, Phan Van Nguyen Duc, Philipp Jund, Phillip Kravtsov, Pooya Davoodi, Pranav Marathe, Putra Manggala, Qingqing Cao, R S Nikhil Krishna, Rajeshwar Reddy T, Ramon ViñAs, Rasmus Diederichsen, Reuben Morais, robert, Rohit Gupta, Roland Zimmermann, Roman Soldatow, RonLek, Ruizhe, Ryan Jiang, saishruthi, Saleem Abdulrasool, Sami Kama, Sana-Damani, sdamani, Sean Morgan, seanshpark, Sebastien Iooss, Serv-Inc, Severen Redwood, Shashank Gupta, shashvat, Shashvat Chand Shahi, Shubham Goyal, Sigrid Keydana, Siju Samuel, sleighsoft, smilu97, Son Tran, sremedios, Srini511, srinivasan.narayanamoorthy, Subin, Sumesh Udayakumaran, Sungmann Cho, sunway513, sxwang, Tae-Hwan Jung, Taehoon Lee, Takeo Sawada, Taylor Jakobson, Ted Chang, TengLu, terryky, ThisIsIsaac, Thomas Deegan, Thomas Hagebols, tianyapiaozi, Till Hoffmann, Tim Zaman, Tongxuan Liu, Trent Lo, Trevor Morris, TungJerry, Tyorden, Uday Bondhugula, v1incent, Vasileios Lioutas, vbvg2008, Vijay Ravichandran, Viktor Gal, Vincent, Vishnuvardhan Janapati, Vivek Suryamurthy, wangsiyu, wateryzephyr, Wei Wang, Wen-Heng (Jack) Chung, wenxizhu, Will Battel, William D. Irons, winstonq, wyzhao, Xiaoming (Jason) Cui, Xiaoquan Kong, Xin, Xinping Wang, Yann-Yy, Yasir Modak, Yasuhiro Matsumoto, Yong Tang, Yongfeng Gu, Yuan (Terry) Tang, Yuchen Ying, zhangyujing, zyeric, 王振华 (Zhenhua Wang), 黄鑫

goldiegadde released this Sep 10, 2020 · 101 commits to r1.15 since this release

Release 1.15.0-rc0

This is the last 1.x release for TensorFlow. We do not expect to update the 1.x branch with features, although we will issue patch releases to fix vulnerabilities for at least one year.

Major Features and Improvements

  • As announced, tensorflow pip package will by default include GPU support (same as tensorflow-gpu now) for the platforms we currently have GPU support (Linux and Windows). It will work on machines with and without Nvidia GPUs. tensorflow-gpu will still be available, and CPU-only packages can be downloaded at tensorflow-cpu for users who are concerned about package size.
  • TensorFlow 1.15 contains a complete implementation of the 2.0 API in its compat.v2 module . It contains a copy of the 1.15 main module (without contrib ) in the compat.v1 module . TensorFlow 1.15 is able to emulate 2.0 behavior using the enable_v2_behavior() function.
    This enables writing forward compatible code: by explicitly importing either tensorflow.compat.v1 or tensorflow.compat.v2 , you can ensure that your code works without modifications against an installation of 1.15 or 2.0.
  • EagerTensor now supports numpy buffer interface for tensors.
  • Add toggles tf.enable_control_flow_v2() and tf.disable_control_flow_v2() for enabling/disabling v2 control flow.
  • Enable v2 control flow as part of tf.enable_v2_behavior() and TF2_BEHAVIOR=1 .
  • AutoGraph translates Python control flow into TensorFlow expressions, allowing users to write regular Python inside tf.function -decorated functions. AutoGraph is also applied in functions used with tf.data , tf.distribute and tf.keras APIS.
  • Adds enable_tensor_equality() , which switches the behavior such that:
    • Tensors are no longer hashable.
    • Tensors can be compared with == and !=, yielding a Boolean Tensor with element-wise comparison results. This will be the default behavior in 2.0.

Breaking Changes

  • Tensorflow code now produces 2 different pip packages: tensorflow_core containing all the code (in the future it will contain only the private implementation) and tensorflow which is a virtual pip package doing forwarding to tensorflow_core (and in the future will contain only the public API of tensorflow). We don’t expect this to be breaking, unless you were importing directly from the implementation.
  • TensorFlow 1.15 is built using devtoolset7 (GCC7) on Ubuntu 16. This may lead to ABI incompatibilities with extensions built against earlier versions of TensorFlow.
  • Deprecated the use of constraint= and .constraint with ResourceVariable.
  • tf.keras :
    • OMP_NUM_THREADS is no longer used by the default Keras config. To configure the number of threads, use tf.config.threading APIs.
    • tf.keras.model.save_model and model.save now defaults to saving a TensorFlow SavedModel.
    • keras.backend.resize_images (and consequently, keras.layers.Upsampling2D ) behavior has changed, a bug in the resizing implementation was fixed.
    • Layers now default to float32 , and automatically cast their inputs to the layer’s dtype. If you had a model that used float64 , it will probably silently use float32 in TensorFlow2, and a warning will be issued that starts with Layer «layer-name» is casting an input tensor from dtype float64 to the layer’s dtype of float32. To fix, either set the default dtype to float64 with tf.keras.backend.set_floatx(‘float64′) , or pass dtype=’float64’ to each of the Layer constructors. See tf.keras.layers.Layer for more information.
    • Some tf.assert_* methods now raise assertions at operation creation time (i.e. when this Python line executes) if the input tensors’ values are known at that time, not during the session.run(). When this happens, a noop is returned and the input tensors are marked non-feedable. In other words, if they are used as keys in feed_dict argument to session.run() , an error will be raised. Also, because some assert ops don’t make it into the graph, the graph structure changes. A different graph can result in different per-op random seeds when they are not given explicitly (most often).

Bug Fixes and Other Changes

  • tf.data :
    • Promoting unbatch from experimental to core API.
    • Adding support for datasets as inputs to from_tensors and from_tensor_slices and batching and unbatching of nested datasets.
  • tf.keras :
    • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format, which allows the saved checkpoints to be compatible with model.load_weights .
    • Saving a Keras Model using tf.saved_model.save now saves the list of variables, trainable variables, regularization losses, and the call function.
    • Deprecated tf.keras.experimental.export_saved_model and tf.keras.experimental.function . Please use tf.keras.models.save_model(. save_format=’tf’) and tf.keras.models.load_model instead.
    • Add an implementation=3 mode for tf.keras.layers.LocallyConnected2D and tf.keras.layers.LocallyConnected1D layers using tf.SparseTensor to store weights, allowing a dramatic speedup for large sparse models.
    • Enable the Keras compile API experimental_run_tf_function flag by default. This flag enables single training/eval/predict execution path. With this 1. All input types are converted to Dataset . 2. When distribution strategy is not specified this goes through the no-op distribution strategy path. 3. Execution is wrapped in tf.function unless run_eagerly=True is set in compile.
    • Raise error if batch_size argument is used when input is dataset/generator/keras sequence.
  • tf.lite
    • Add GATHER support to NN API delegate.
    • tflite object detection script has a debug mode.
    • Add delegate support for QUANTIZE .
    • Added evaluation script for COCO minival.
    • Add delegate support for QUANTIZED_16BIT_LSTM .
    • Converts hardswish subgraphs into atomic ops.
  • Add support for defaulting the value of cycle_length argument of tf.data.Dataset.interleave to the number of schedulable CPU cores.
  • parallel_for : Add converter for MatrixDiag .
  • Add narrow_range attribute to QuantizeAndDequantizeV2 and V3.
  • Added new op: tf.strings.unsorted_segment_join .
  • Add HW acceleration support for topK_v2 .
  • Add new TypeSpec classes.
  • CloudBigtable version updated to v0.10.0.
  • Expose Head as public API.
  • Update docstring for gather to properly describe the non-empty batch_dims case.
  • Added tf.sparse.from_dense utility function.
  • Improved ragged tensor support in TensorFlowTestCase .
  • Makes the a-normal form transformation in Pyct configurable as to which nodes are converted to variables and which are not.
  • ResizeInputTensor now works for all delegates.
  • Add EXPAND_DIMS support to NN API delegate TEST: expand_dims_test
  • tf.cond emits a StatelessIf op if the branch functions are stateless and do not touch any resources.
  • tf.cond , tf.while and if and while in AutoGraph now accept a nonscalar predicate if has a single element. This does not affect non-V2 control flow.
  • tf.while_loop emits a StatelessWhile op if the cond and body functions are stateless and do not touch any resources.
  • Refactors code in Quant8 LSTM support to reduce TFLite binary size.
  • Add support of local soft device placement for eager op.
  • Add HW acceleration support for LogSoftMax .
  • Added a function nested_value_rowids for ragged tensors.
  • Add guard to avo >tf.math.cumulative_logsumexp operation .
  • Add tf.ragged.stack .
  • Fix memory allocation problem when calling AddNewInputConstantTensor .
  • Delegate application failure leaves interpreter in valid state.
  • Add check for correct memory alignment to MemoryAllocation::MemoryAllocation() .
  • Extracts NNAPIDelegateKernel from nnapi_delegate.cc
  • Added support for FusedBatchNormV3 in converter.
  • A ragged to dense op for directly calculating tensors.
  • Fix accidental quadratic graph construction cost in graph-mode tf.gradients() .

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

a6802739, Aaron Ma, Abdullah Selek, Abolfazl Shahbazi, Ag Ramesh, Albert Z. Guo, Albin Joy, Alex Itkes, Alex Sergeev, Alexander Pivovarov, Alexey Romanov, alhkad, Amit Srivastava, amoitra, Andrew Lihonosov, Andrii Prymostka, Anuj Rawat, Astropeak, Ayush Agrawal, Bairen Yi, Bas Aarts, Bastian Eichenberger, Ben Barsdell, Benjamin Peterson, bhack, Bharat Raghunathan, Bhavani Subramanian, Bryan Cutler, candy.dc, Cao Zongyan, Captain-Pool, Casper Da Costa-Luis, Chen Guoyin, Cheng Chang, chengchingwen, Chong Yan, Choong Yin Thong, Christopher Yeh, Clayne Robison, Coady, Patrick, Dan Ganea, David Norman, Denis Khalikov, Deven Desai, Diego Caballero, Duncan Dean, Duncan Riach, Dwight J Lyle, Eamon Ito-Fisher, eashtian3, EFanZh, ejot, Elroy Ashtian Jr, Eric Schweitz, Fangjun Kuang, Fei Hu, fo40225, formath, Fred Reiss, Frederic Bastien, Fredrik Knutsson, G. Hussain Chinoy, Gabriel, gehring, George Grzegorz Pawelczak, Gianluca Varisco, Gleb Popov, Greg Peatfield, Guillaume Klein, Gurpreet Singh, Gustavo Lima Chaves, haison, Haraldur TóMas HallgríMsson, HarikrishnanBalagopal, HåKon Sandsmark, I-Hong, Ilham Firdausi Putra, Imran Salam, Jason Zaman, Jason Zavaglia, jayhpark530, jefby, Jeff Daily, Jeffrey Poznanovic, Jekyll Lai, Jeroen BéDorf, Jerry Shih, jerryyin, jiakai, JiangXIAO, Joe Bowser, Joel Shapiro, Johan Gunnarsson, Jojimon Varghese, Joon, Josh Beal, Julian Niedermeier, Jun Wan, Junqin Zhang, Junyuan Xie, Justin Tunis, Kaixi Hou, Karl Lessard, Karthik Muthuraman, Kbhute-Ibm, khanhlvg, Koock Yoon, kstuedem, Kyuwon Kim, Lakshay Tokas, leike666666, leonard951, Leslie-Fang, Leslie-Fang-Intel, Li, Guizi, Lukas Folle, Lukas Geiger, Mahmoud Abuzaina, Manraj Singh Grover, Margaret Maynard-Reid, Mark Ryan, Matt Conley, Matthew Bentham, Matthew Denton, mbhuiyan, mdfaijul, Mei Jie, merturl, MichaelKonobeev, Michal W. Tarnowski, minds, mpppk, musikisomorphie, Nagy Mostafa, Nayana Thorat, Neil, Niels Ole Salscheider, Niklas SilfverströM, Niranjan Hasabnis, ocjosen, olramde, Pariksheet Pinjari, Patrick J. Lopresti, Patrik Gustavsson, per1234, PeterLee, Phan Van Nguyen Duc, Phillip Kravtsov, Pooya Davoodi, Pranav Marathe, Putra Manggala, Qingqing Cao, Rajeshwar Reddy T, Ramon ViñAs, Rasmus Diederichsen, Reuben Morais, richardbrks, robert, RonLek, Ryan Jiang, saishruthi, Saket Khandelwal, Saleem Abdulrasool, Sami Kama, Sana-Damani, Sergii Khomenko, Severen Redwood, Shubham Goyal, Sigrid Keydana, Siju Samuel, sleighsoft, smilu97, Son Tran, Srini511, srinivasan.narayanamoorthy, Sumesh Udayakumaran, Sungmann Cho, Tae-Hwan Jung, Taehoon Lee, Takeshi Watanabe, TengLu, terryky, TheMindVirus, ThisIsIsaac, Till Hoffmann, Timothy Liu, Tomer Gafner, Tongxuan Liu, Trent Lo, Trevor Morris, Uday Bondhugula, Vasileios Lioutas, vbvg2008, Vishnuvardhan Janapati, Vivek Suryamurthy, Wei Wang, Wen-Heng (Jack) Chung, wenxizhu, William D. Irons, winstonq, wyzhao, Xiaoming (Jason) Cui, Xinan Jiang, Xinping Wang, Yann-Yy, Yasir Modak, Yong Tang, Yongfeng Gu, Yuchen Ying, Yuxin Wu, zyeric, 王振华 (Zhenhua Wang)

goldiegadde released this Aug 23, 2020 · 197 commits to r2.0 since this release

Release 2.0.0-rc0

Major Features and Improvements

TensorFlow 2.0 focuses on simplicity and ease of use, featuring updates like:

  • Easy model building with Keras and eager execution.
  • Robust model deployment in production on any platform.
  • Powerful experimentation for research.
  • API simplification by reducing duplication and removing deprecated endpoints.

For details on best practices with 2.0, see the Effective 2.0 guide

For information on upgrading your existing TensorFlow 1.x models, please refer to our Upgrade and Migration guides. We have also released a collection of tutorials and getting started guides.

Highlights

  • TF 2.0 delivers Keras as the central high level API used to build and train models. Keras provides several model-building APIs such as Sequential, Functional, and Subclassing along with eager execution, for immediate iteration and intuitive debugging, and tf.data, for building scalable input pipelines. Checkout guide for additional details.
  • Distribution Strategy: TF 2.0 users will be able to use the tf.distribute.Strategy API to distribute training with minimal code changes, yielding great out-of-the-box performance. It supports distributed training with Keras model.fit, as well as with custom training loops. Multi-GPU support is available, along with experimental support for multi worker and Cloud TPUs. Check out the guide for more details.
  • Functions, not Sessions. The traditional declarative programming model of building a graph and executing it via a tf.Session is discouraged, and replaced with by writing regular Python functions. Using the tf.function decorator, such functions can be turned into graphs which can be executed remotely, serialized, and optimized for performance.
  • Unification of tf.train.Optimizers and tf.keras.Optimizers. Use tf.keras.Optimizers for TF2.0. compute_gradients is removed as public API, and use GradientTape to compute gradients.
  • AutoGraph translates Python control flow into TensorFlow expressions, allowing users to write regular Python inside tf.function -decorated functions. AutoGraph is also applied in functions used with tf.data, tf.distribute and tf.keras APIs.
  • Unification of exchange formats to SavedModel. All TensorFlow ecosystem projects (TensorFlow Lite, TensorFlow JS, TensorFlow Serving, TensorFlow Hub) accept SavedModels. Model state should be saved to and restored from SavedModels.
  • API Changes: Many API symbols have been renamed or removed, and argument names have changed. Many of these changes are motivated by consistency and clarity. The 1.x API remains available in the compat.v1 module. A list of all symbol changes can be found here.
  • API clean-up, included removing tf.app , tf.flags , and tf.logging in favor of absl-py.
  • No more global variables with helper methods like tf.global_variables_initializer and tf.get_global_step .

Breaking Changes

  • Many backwards incompatible API changes have been made to clean up the APIs and make them more consistent.
  • tf.contrib has been deprecated, and functionality has been either migrated to the core TensorFlow API, to an ecosystem project such as tensorflow/addons or tensorflow/io, or removed entirely.
  • Premade estimators in the tf.estimator.DNN/Linear/DNNLinearCombined family have been updated to use tf.keras.optimizers instead of the tf.compat.v1.train.Optimizer s. If you do not pass in an optimizer= arg or if you use a string, the premade estimator will use the Keras optimizer. This is checkpoint breaking, as the optimizers have separate variables. A checkpoint converter tool for converting optimizers is included with the release, but if you want to avoid any change, switch to the v1 version of the estimator: tf.compat.v1.estimator.DNN/Linear/DNNLinearCombined* .
  • The equality operation on Tensors & Variables now compares on value instead of id() . As a result, both Tensors & Variables are no longer hashable types.
  • Layers now default to float32, and automatically cast their inputs to the layer’s dtype. If you had a model that used float64, it will probably silently use float32 in TensorFlow 2, and a warning will be issued that starts with «Layer is casting an input tensor from dtype float64 to the layer’s dtype of float32». To fix, either set the default dtype to float64 with tf.keras.backend.set_floatx(‘float64′) , or pass dtype=’float64’ to each of the Layer constructors. See tf.keras.layers.Layer for more information.

Refer to our public project status tracker and issues tagged with 2.0 on GitHub for insight into recent issues and development progress.

If you experience any snags when using TF 2.0, please let us know at the TF 2.0 Testing User Group. We have a support mailing list as well as weekly testing meetings, and would love to hear your migration feedback and questions.

Bug Fixes and Other Changes

  • Add support for TensorArrays to tf.data Dataset .
  • Integrate Ragged Tensors with tf.data .
  • All core and experimental tf.data transformations that input user-defined functions can span multiple devices now.
  • Extending the TF 2.0 support for shuffle(. reshuffle_each_iteration=True) and cache() to work across different Python iterators for the same dataset.
  • Removing the experimental_numa_aware option from tf.data.Options .
  • Add num_parallel_reads and passing in a Dataset containing filenames into TextLineDataset and FixedLengthRecordDataset .
  • Add support for defaulting the value of cycle_length argument of tf.data.Dataset.interleave to the number of schedulable CPU cores.
  • Promoting tf.data.experimental.enumerate_dataset to core as tf.data.Dataset.enumerate .
  • Promoting tf.data.experimental.unbatch to core as tf.data.Dataset.unbatch .
  • Adds option for introducing slack in the pipeline to reduce CPU contention, via tf.data.Options().experimental_slack = True
  • Added experimental support for parallel batching to batch() and padded_batch() . This functionality can be enabled through tf.data.Options()
  • Support cancellation of long-running reduce .
  • Now we use dataset node name as prefix instead of the op name, to identify the component correctly in metrics, for pipelines with repeated components.
  • Enable tf.distribute.experimental.MultiWorkerMirroredStrategy working in eager mode.
  • Disable run_eagerly and distribution strategy if there are symbolic tensors added to the model using add_metric or add_loss .
  • Bug fix: loss and gradients should now more reliably be correctly scaled w.r.t. the global batch size when using a tf.distribute.Strategy .
  • Set default loss reduction as AUTO for improving reliability of loss scaling with distribution strategy and custom training loops. AUTO indicates that the reduction option will be determined by the usage context. For almost all cases this defaults to SUM_OVER_BATCH_SIZE . When used in distribution strategy scope, outside of built-in training loops such as tf.keras compile and fit , we expect reduction value to be ‘None’ or ‘SUM’. Using other values will raise an error.
  • Support for multi-host ncclAllReduce in Distribution Strategy.
  • Replace tf.contrib.estimator.add_metrics with tf.estimator.add_metrics
  • Use tf.compat.v1.estimator.inputs instead of tf.estimator.inputs
  • Replace contrib references with tf.estimator.experimental.* for apis in early_s in Estimator
  • Canned Estimators will now use keras optimizers by default. An error will be raised if tf.train.Optimizers are used, and you will have to switch to tf.keras.optimizers or tf.compat.v1 canned Estimators.
  • A checkpoint converter for canned Estimators has been provided to transition canned Estimators that are warm started from tf.train.Optimizers to tf.keras.optimizers.
  • Default aggregation for canned Estimators is now SUM_OVER_BATCH_SIZE. To maintain previous default behavior, please pass SUM as the loss aggregation method.
  • Canned Estimators don’t support input_layer_partitioner arg in the API. If you have this arg, you will have to switch to tf.compat.v1 canned Estimators.
  • Estimator.export_savedmodel has been renamed export_saved_model
  • When saving to SavedModel, Estimators will strip default op attributes. This is almost always the correct behavior, as it is more forwards compatible, but if you require that default attributes are saved with the model, please use tf.compat.v1.Estimator
  • Feature Columns have been upgraded to be more Eager-friendly and to work with Keras. As a result, tf.feature_column.input_layer has been deprecated in favor of tf.keras.layers.DenseFeatures. v1 feature columns have direct analogues in v2 except for shared_embedding_columns, which are not cross-compatible with v1 and v2. Use tf.feature_column.shared_embeddings instead.
  • Losses are scaled in canned estimator v2 and not in the optimizers anymore. If you are using Estimator + distribution strategy + optimikzer v1 then the behavior does not change. This implies that if you are using custom estimator with optimizer v2, you have to scale losses. We have new utilities to help scale losses tf.nn.compute_average_loss , tf.nn.scale_regularization_loss .
  • Premade models (including Linear and WideDeep) have been introduced for the purpose of replacing Premade estimators.
  • Model saving changes
  • model.save and tf.saved_model.save may now save to the TensorFlow SavedModel format. The model can be restored using tf.keras.models.load_model . HDF5 files are still supported, and may be used by specifying save_format=»h5″ when saving.
  • tf.keras.model.save_model and model.save now defaults to saving a TensorFlow SavedModel. HDF5 files are still supported.
  • Deprecated tf.keras.experimental.export_saved_model and tf.keras.experimental.function . Please use tf.keras.models.save_model(. save_format=’tf’) and tf.keras.models.load_model` instead.
  • Raw TensorFlow functions can now be used in conjunction with the Keras Functional API during model creation. This obviates the need for users to create Lambda layers in most cases when using the Functional API. Like Lambda layers, TensorFlow functions that result in Variable creation or assign ops are not supported.
  • Add support for passing list of lists to the metrics argument in Keras `compile.
  • Add tf.keras.layers.AbstractRNNCell as the preferred implementation for RNN cells in TF v2. User can use it to implement RNN cells with custom behavior.
  • Keras training and validation curves are shown on the same plot when using the TensorBoard callback.
  • Switched Keras fit/evaluate/predict execution to use only a single unified path by default unless eager execution has been explicitly disabled, regardless of input type. This unified path places an eager-friendly training step inside of a tf.function . With this 1. All input types are converted to Dataset . 2. The path assumes there is always a distribution strategy. when distribution strategy is not specified the path uses a no-op distribution strategy. 3. The training step is wrapped in tf.function unless run_eagerly=True is set in compile. The single path execution code does not yet support all use cases. We fallback to the existing v1 execution paths if your model contains the following: 1. sample_weight_mode in compile 2. weighted_metrics in compile 3. v1 optimizer 4. target tensors in compile. If you are experiencing any issues because of this change, please inform us (file an issue) about your use case and you can unblock yourself by setting experimental_run_tf_function=False in compile meanwhile. We have seen couple of use cases where the model usage pattern is not as expected and would not work with this change. 1. output tensors of one layer is used in the constructor of another. 2. symbolic tensors outside the scope of the model are used in custom loss functions. The flag can be disabled for these cases and ideally the usage pattern will need to be fixed.
  • OMP_NUM_THREADS is no longer used by the default Keras config. To configure the number of threads, use tf.config.threading APIs.
  • Mark Keras set_session as compat.v1 only.
  • tf.keras.estimator.model_to_estimator now supports exporting to tf.train.Checkpoint format , which allows the saved checkpoints to be compatible with model.load_weights .
  • keras.backend.resize_images (and consequently, keras.layers.Upsampling2D ) behavior has changed, a bug in the resizing implementation was fixed.
  • Add an implementation=3 mode for tf.keras.layers.LocallyConnected2D and tf.keras.layers.LocallyConnected1D layers using tf.SparseTensor to store weights, allowing a dramatic speedup for large sparse models.
  • Raise error if batch_size argument is used when input is dataset/generator/keras sequence.
  • Update TF 2.0 keras.backend.name_scope to use TF 2.0 name_scope .
  • Add v2 module aliases for losses, metrics, initializers and optimizers: tf.losses = tf.keras.losses & tf.metrics = tf.keras.metrics & tf.initializers = tf.keras.initializers & tf.optimizers = tf.keras.optimizers .
  • Updates binary cross entropy logic in Keras when input is probabilities. Instead of converting probabilities to logits, we are using the cross entropy formula for probabilities.
  • Added public APIs for cumsum and cumprod keras backend functions.
  • Add support for temporal sample weight mode in subclassed models.
  • Raise ValueError if an integer is passed to the training APIs.
  • Added fault-tolerance support for training Keras model via model.fit() with MultiWorkerMirroredStrategy , tutorial available.
  • Callbacks are supported in MultiWorkerMirroredStrategy .
  • Custom Callback tutorial is now available.
  • To train with tf.distribute , Keras api is recommended over estimator.
  • steps_per_epoch and steps arguments are supported with numpy arrays.
  • New error message when unexpected keys are used in sample_weight/class_weight dictionaries
  • Losses are scaled in Keras compile/fit and not in the optimizers anymore. If you are using custom training loop, we have new utilities to help scale losses tf.nn.compute_average_loss , tf.nn.scale_regularization_loss .
  • Layer apply and add_variable APIs are deprecated.
  • Added support for channels first data format in cross entropy losses with logits and support for tensors with unknown ranks.
  • Error messages will be raised if add_update , add_metric , add_loss , activity regularizers are used inside of a control flow branch.
  • New loss reduction types: 1. AUTO : Indicates that the reduction option will be determined by the usage context. For almost all cases this defaults to SUM_OVER_BATCH_SIZE . When used with tf.distribute.Strategy , outside of built-in training loops such as tf.keras compile and fit , we expect reduction value to be SUM or NONE . Using AUTO in that case will raise an error. 2. NONE : Weighted losses with one dimension reduced (axis=-1, or axis specified by loss function). When this reduction type used with built-in Keras training loops like fit / evaluate , the unreduced vector loss is passed to the optimizer but the reported loss will be a scalar value. 3. SUM : Scalar sum of weighted losses. 4. SUM_OVER_BATCH_SIZE : Scalar SUM divided by number of elements in losses. This reduction type is not supported when used with tf.distribute.Strategy outside of built-in training loops like tf.keras compile / fit .
  • Added support for TFLiteConverter Python API in 2.0. Contains functions from_saved_model, from_keras_file, and from_concrete_functions.
  • Removed lite.OpHint , lite.experimental , and lite.constant from 2.0 API.
  • Added support for tflite_convert command line tool in 2.0.
  • Post-training quantization tool supports quantizing weights shared by multiple operations. The models made with versions of this tool will use INT8 types for weights and will only be executable interpreters from this version onwards.
  • Post-training quantization tool supports fp16 weights and GPU delegate acceleration for fp16.
  • Expose tf.contrib.proto.* ops in tf.io (they will exist in TF2)
  • Remove tf.contrib.timeseries dependency on TF distributions.
  • Replace contrib references with tf.estimator.experimental.* for apis in early_stopping.py
  • Bug fix for tf.tile gradient .
  • TF code now resides in tensorflow_core and tensorflow is just a virtual pip package. No code changes are needed for projects using TensorFlow, the change is transparent
  • Added gradient for SparseToDense op.
  • Expose a flag that allows the number of threads to vary across Python benchmarks.
  • ResourceVariable’s gather op supports batch dimensions.
  • image.resize in 2.0 now supports gradients for the new resize kernels.
  • removed tf.string_split from v2 API
  • Variadic reduce is supported on CPU Variadic reduce is supported on CPU
  • Added GPU implementation of tf.linalg.tridiagonal_solve .
  • Delete unused lookup table code
  • Remove unused StringViewVariantWrapper .
  • Delete unused Fingerprint64Map op registration
  • Add broadcasting support to tf.matmul .
  • Add ellipsis (. ) support for tf.einsum() .
  • ResourceVariable support for gather_nd .
  • Add expand_composites argument to all nest.* methods.
  • Standardize the LayerNormalization API by replacing the args norm_axis and params_axis with axis .
  • Add a new «result_type» parameter to tf.strings.split
  • add_update can now be passed a zero-arg callable in order to support turning off the update when setting trainable=False on a Layer of a Model compiled with run_eagerly=True .
  • Added tf.random.binomial .
  • Extend tf.function with basic support for CompositeTensors arguments (such as SparseTensor and RaggedTensor).
  • Add name argument to tf.string_split and tf.strings_split .
  • Added strings.byte_split .
  • CUDNN_INSTALL_PATH, TENSORRT_INSTALL_PATH, NCCL_INSTALL_PATH, NCCL_HDR_PATH are deprecated. Use TF_CUDA_PATHS instead which supports a comma-separated list of base paths that are searched to find CUDA libraries and headers.
  • Add RaggedTensor.placeholder() .
  • Add pfor converter for Squeeze .
  • Renamed tf.image functions to remove duplicate «image» where it is redundant.
  • Add C++ Gradient for BatchMatMulV2.
  • parallel_for.pfor : add converters for Softmax, LogSoftmax, IsNaN, All, Any, and MatrixSetDiag.
  • parallel_for : add converters for LowerTriangularSolve and Cholesky.
  • Add ragged tensor support to tf.squeeze .
  • Allow LinearOperator.solve to take a LinearOperator .
  • Allow all dtypes for LinearOperatorCirculant .
  • Introduce MaxParallelism method
  • parallel_for : add converter for BroadcastTo .
  • Add LinearOperatorHouseholder .
  • Added key and skip methods to random.experimental.Generator .
  • Adds Philox support to new stateful RNG’s XLA path.
  • Update RaggedTensors to support int32 row_splits.
  • Add TensorSpec support for CompositeTensors.
  • Added partial_pivoting input parameter to tf.linalg.tridiagonal_solve .
  • Extend tf.strings.split to support inputs with any rank
  • Removing the experimental_numa_aware option from tf.data.Options .
  • Improve the performance of datasets using from_tensors() .
  • Add tf.linalg.tridiagonal_mul op .
  • Add LinearOperatorToeplitz .
  • Added gradient to tf.linalg.tridiagonal_solve .
  • Upgraded LIBXSMM to version 1.11.
  • parallel_for : add converters for LogMatrixDeterminant and MatrixBandPart .
  • Uniform processing of quantized embeddings by Gather and EmbeddingLookup Ops
  • Correct a misstatement in the documentation of the sparse softmax cross entropy logit parameter.
  • parallel_for : Add converters for OneHot , LowerBound , UpperBound .
  • Added GPU implementation of tf.linalg.tridiagonal_matmul .
  • Add gradient to tf.linalg.tridiagonal_matmul .
  • Add tf.ragged.boolean_mask .
  • tf.switch_case added, which selects a branch_fn based on a branch_index.
  • The C++ kernel of gather op supports batch dimensions.
  • Promoting unbatch from experimental to core API.
  • Fixed default value and documentation for trainable arg of tf.Variable.
  • Adds tf.enable_control_flow_v2() and tf.disable_control_flow_v2() .
  • EagerTensor now supports buffer interface for tensors.
  • This change bumps the version number of the FullyConnected Op to 5.
  • tensorflow : crash when pointer become nullptr.
  • parallel_for : Add converter for MatrixDiag .
  • Add ‘narrow_range’ attribute to QuantizeAndDequantizeV2 and V3.
  • Added new op: tf.strings.unsorted_segment_join .
  • Tensorflow code now produces 2 different pip packages: tensorflow_core containing all the code (in the future it will contain only the private implementation) and tensorflow which is a virtual pip package doing forwarding to tensorflow_core (and in the future will contain only the public API of tensorflow)
  • Adding support for datasets as inputs to from_tensors and from_tensor_slices and batching and unbatching of nested datasets.
  • Add HW acceleration support for topK_v2
  • Add new TypeSpec classes
  • CloudBigtable version updated to v0.10.0 BEGIN_PUBLIC CloudBigtable version updated to v0.10.0
  • Deprecated the use of constraint= and .constraint with ResourceVariable.
  • Expose Head as public API.
  • Update docstring for gather to properly describe the non-empty batch_dims case.
  • Added tf.sparse.from_dense utility function.
  • Add GATHER support to NN API delegate
  • Improved ragged tensor support in TensorFlowTestCase .
  • Makes the a-normal form transformation in Pyct configurable as to which nodes are converted to variables and which are not.
  • ResizeInputTensor now works for all delegates
  • Start of open development of TF, TFLite, XLA MLIR dialects.
  • Add EXPAND_DIMS support to NN API delegate TEST: expand_dims_test
  • tf.cond emits a StatelessIf op if the branch functions are stateless and do not touch any resources.
  • Add support of local soft device placement for eager op.
  • Pass partial_pivoting to the _TridiagonalSolveGrad .
  • Add HW acceleration support for LogSoftMax
  • Added a function nested_value_row >QUANTIZE
  • add tf.math.cumulative_logsumexp operation.
  • Add tf.ragged.stack .
  • Add delegate support for QUANTIZED_16BIT_LSTM .
  • tf.while_loop emits a StatelessWhile op if the cond and body functions are stateless and do not touch any resources.
  • Fix memory allocation problem when calling AddNewInputConstantTensor .
  • Delegate application failure leaves interpreter in val >dynamic constructor argument in Layer and Model, which should be set to True when using imperative control flow in the call method.
  • ResourceVariable and Variable no longer accepts constraint in the constructor, nor expose it as a @Property.
  • Add UnifiedGRU as the new GRU implementation for tf2.0. Change the default recurrent activation function for GRU from ‘hard_sigmo , reset_after=False) to fallback to 1.x behavior.
  • Begin adding Go wrapper for C Eager API
  • XLA HLO graphs can be inspected with interactive_graphviz tool now.
  • Add dataset ops to the graph (or create kernels in Eager execution) during the python Dataset object creation instead doing it during Iterator creation time.
  • Add batch_dims argument to tf.gather.
  • Removing of dtype in the constructor of initializers and partition_info in call.
  • Add tf.math.nextafter op.
  • Turn on MKL-DNN contraction kernels by default. MKL-DNN dynamically dispatches the best kernel implementation based on CPU vector architecture. To disable them, build with —define=tensorflow_mkldnn_contraction_kernel=0.
  • tf.linspace(start, stop, num) now always uses «stop» as last value (for num > 1)
  • Added top-k to precision and recall to keras metrics.
  • Add a ragged size op and register it to the op dispatcher
  • Transitive dependencies on :pooling_ops were removed. Some users may need to add explicit dependencies on :pooling_ops if they reference the operators from that library.
  • Add CompositeTensor base class.
  • Malformed gif images could result in an access out of bounds in the color palette of the frame. This has been fixed now
  • Add templates and interfaces for creating lookup tables
  • Tensor::UnsafeCopyFromInternal deprecated in favor Tensor::BitcastFrom
  • In map_vectorization optimization, reduce the degree of parallelism in the vectorized map node.
  • Add variant wrapper for absl::string_view
  • Wraps losses passed to the compile API (strings and v1 losses) which are not instances of v2 Loss class in LossWrapper class. => All losses will now use SUM_OVER_BATCH_SIZE reduction as default.
  • Add OpKernels for some stateless maps
  • Add v2 APIs for AUCCurve and AUCSummationMethod enums. #tf-metrics-convergence
  • Allow non-Tensors through v2 losses.
  • Add v2 sparse categorical crossentropy metric. GITHUB_PR_OR_BUG=b/123431691
  • DType is no longer convertible to an int. Use dtype.as_datatype_enum instead of int(dtype) to get the same result.
  • Support both binary and -1/1 label input in v2 hinge and squared hinge losses.
  • Added LinearOperator.adjoint and LinearOperator.H (alias).
  • Expose CriticalSection in core as tf.CriticalSection .
  • Enhanced graphviz output.
  • The behavior of tf.gather is now correct when axis=None and batch_dims tf.linalg.tridiagonal_solve op.
  • Add opkernel templates for common table operations.
  • Fix issue: Callbacks do not log values in eager mode when a deferred build model is used.
  • SignatureDef util functions have been deprecated.
  • Update Fingerprint64Map to use aliases
  • Add legacy string flat hash map op kernels
  • Fix: model.add_loss(symbolic_tensor) should work in ambient eager.
  • Adding clear_losses API to be able to clear losses at the end of forward pass in a custom training loop in eager. GITHUB_PR_OR_BUG=b/123431691
  • Add support for add_metric in the graph function mode. GITHUB_PR_OR_BUG=tf_only
  • Updating cosine similarity loss — removed the negate sign from cosine similarity. GITHUB_PR_OR_BUG=b/123431691
  • TF 2.0 — Update metric name to always reflect what the user has given in compile. Affects following cases 1. When name is given as ‘accuracy’/’crossentropy’ 2. When an aliased function name is used eg. ‘mse’ 3. Removing the weighted prefix from weighted metric names.
  • Workaround for compiler bug(?)
  • Changed default for gradient accumulation for TPU embeddings to true.
  • Adds summary trace API for collecting graph and profile information.
  • image.resize now considers proper pixel centers and has new kernels (incl. anti-aliasing).

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Что нового в TensorFlow 2.0

i.Gorschal twitter logo

Aug 30 ・4 min read

TensorFlow как никто другой оказывает влияние на machine learning комьюнити. Фреймворк с недавних пор уже в бете с литерой 2.0. Это достаточно серьезный апдейт и в этом тексте хочу осветить наиболее значимые моменты.

Это не руководство по AI и для понимания вы уже должны быть знакомы с TensorFlow.

Установка и демонстрационный набор данных

Используем Jupyter Notebook:

Если необходимо использовать графический процессор (CUDA):

Нам также будет необходим тестовый набор данных. Будем использовать набор данных Adult из архива UCI.

Набор данных представляет собой задачу двоичной классификации, которая заключается в прогнозировании, будет ли человек зарабатывать более 50 000 долларов в год.

Для начала проведем базовую предварительную обработку данных, а затем настроим разделение данных в соотношении 80:20:

Если TensorFlow у нас установился, датасет загрузился, то можно переходить к самому интересному.

1. Eager execution по умолчанию

В TensorFlow 2.0 больше не нужно создавать сессию и запускать в нем граф вычислений. Eager execution включено по умолчанию в версии 2.0, так что вы можете создавать модели и запускать их мгновенно. Отключить eager execution можно следующим образом:

2. tf.function и AutoGraph

В то время как eager execution позволяет вам работать с императивным программированием, но как только речь заходит о распределенном обучении, полномасштабной оптимизации, производственные среды style graph execution TensorFlow 1.x имеет свои преимущества по сравнению с eager execution. В TensorFlow 2.0 вы сохраняете выполнение на основе графов, но более гибким способом. Это можно сделать с помощью tf.function и AutoGraph.

tf.function позволяет определять графы TensorFlow с синтаксисом в стиле Python с помощью функции AutoGraph. AutoGraph поддерживает широкий диапазон совместимости с Python, включая оператор if, цикл for, цикл while, итераторы и т. д. Однако существуют ограничения. Здесь вы можете ознакомиться с полным списком доступных на данный момент операторов. Ниже приведен пример, показывающий, как легко определить граф TensorFlow с помощью простого декоратора.

Обратите внимание, что вам не нужно было создавать сессии или поля для заполнения для запуска функции single_layer (). Это одна из замечательных функций tf.function. Из-за этого он выполняет все необходимые оптимизации, чтобы код работал быстрее.

3. tf.variable_scope больше не нужен

В версии TensorFlow 1.x, для того, чтобы использовать tf.layers в качестве переменных и использовать их повторно, Вам нужно был использовать блок tf.variable. Но этого больше не нужно делать в TensorFlow 2.0. Из-за присутствия keras в качестве центрального высокоуровневого API в TensorFlow 2.0 все слои, созданные с использованием tf.layers, могут быть легко помещены в определение tf.keras.Sequential. Это значительно облегчает чтение кода, а также позволяет отслеживать переменные и потери.

В приведенном выше примере вы передали набор данных для обучения через model просто для получения необработанных выходных вероятностей. Обратите внимание, что это просто прямой проход. Вы можете, конечно, пойти вперед и обучать свою модель:

Вы можете получить список обучаемых параметров модели по слоям, например:

4. Кастомные слои это просто

В исследованиях в области машинного обучения и в промышленных приложениях часто возникает необходимость в написании кастомных слоев для удовлетворения конкретных случаев использования. TensorFlow 2.0 позволяет очень просто написать собственный слой и использовать его вместе с существующими слоями. Вы также можете настроить прямой проход вашей модели любым удобным для вас способом.

Чтобы создать кастомный слой, проще всего расширить класс Layer из tf.keras.layers, а затем определить его соответствующим образом. Вы создадите кастомный слой, а затем определите его прямые вычисления.

Можно создать несколько слоев, расширяя класс Model из tf.keras. Подробнее о создании моделей можно узнать здесь.

5. Гибкость в обучении модели

TensorFlow может использовать автоматическое дифференцирование для вычисления градиентов функции потерь относительно параметров модели. tf.GradientTape создает ленту в контексте, который используется TensorFlow для отслеживания градиентов, записанных при каждом вычислении на этой ленте. Чтобы понять это, давайте определим модель на более низком уровне, расширив класс tf.keras.Model.

Обратите внимание, что топология этой модели точно такая же, как и та, которую вы определили ранее. Чтобы обучить эту модель с использованием автоматического дифференцирования, вам нужно по-разному определить функцию потерь и оптимайзер:

Теперь вы определите метрики, которые будут использоваться для измерения производительности сети, включающей ее обучение. Под производительностью здесь подразумеваются потеря и точность модели.

tf.data предоставляет служебные методы для определения конвейеров входных данных. Это особенно полезно, когда вы имеете дело с большим объемом данных.

Теперь вы определите генератор данных, который будет генерировать пакеты данных во время обучения модели.

Теперь вы готовы обучать модель, используя tf.GradientTape. Во-первых, вы определите метод, который будет тренировать модель с данными, которые вы только что определили, используя tf.data.DataSet. Вы также оберните этапы обучения модели с помощью декоратора tf.function, чтобы ускорить вычисления.

6. Наборы данных TensorFlow

Отдельный модуль DataSets используется для отточенной работы с сетевой моделью. Вы уже видели это в предыдущем примере. В этом разделе вы увидите, как можно загрузить набор данных MNIST именно так, как вам нужно.

Вы можете установить библиотеку tenorflow_datasets с помощью pip. Как только он установлен, вы готовы приступать к работе. Он предоставляет несколько вспомогательных функций, которые помогут вам гибко подготовить конвейер построения набора данных. Вы можете узнать больше об этих функциях здесь и здесь. Теперь вы увидите, как вы можете построить конвейер ввода данных для загрузки в наборе данных MNIST.

Вы можете игнорировать предупреждение. Обратите внимание, как оригинально tennsflow_datasets обрабатывает конвейер.

Теперь вы можете отобразить первое изображение из коллекции изображений, в которую вы загрузили. Обратите внимание, что tennsflow_datasets работает в активном режиме и настроен на основе графа.

7. Политика автоматической смешанной точности

Политика смешанной точности была предложена NVIDIA в прошлом году. Оригинал статьи находится здесь. Кратко об основной идеи политики смешанной точности: использовать смесь половин (FP16) и полной точности (FP32) и использовать преимущества обоих миров. Она показала потрясающие результаты в обучении очень глубоких нейронных сетей (как с точки зрения времени и оценки).

Если вы работаете в среде графического процессора с поддержкой CUDA (например, Volta Generation, Tesla T4) и установили вариант TensorFlow 2.0 с графическим процессором, вы можете дать TensorFlow команду обучаться со смешанной точностью, например, так:

Это автоматически распределит операции графа TensorFlow соответственно. Вы сможете увидеть хороший скачок производительности вашей модели. Вы также можете оптимизировать основные операции TensorFlow с помощью политики смешанной точности. Подробнее почитайте в этой статье.

8. Распределенное обучение

TensorFlow 2.0 позволяет очень легко распределять процесс обучения по нескольким графическим процессорам. Это особенно полезно для производственных целей, когда вам приходится встречать сверхтяжелые нагрузки. Это так же просто, как поместить ваш тренировочный блок в блок with.

Сначала вы указываете стратегию распределения следующим образом:

Зеркальная стратегия создает одну реплику для каждого графического процессора, а переменные модели одинаково отражаются в разных графических процессорах. Теперь вы можете использовать определенную стратегию следующим образом:

Обратите внимание, что приведенный выше фрагмент кода будет полезен только в том случае, если в одной системе настроено несколько графических процессоров. Существует несколько стратегий распределения, которые вы можете настроить. Подробнее об этом можно прочитать здесь.

9. TensorBoard в Jupyter notebook

Это, пожалуй, самая захватывающая часть этого апдейта. Вы можете визуализировать обучение модели непосредственно в Jupyter notebook через TensorBoard. Новый TensorBoard имеет множество интересных функций, таких как профилирование памяти, просмотр данных изображения, включая матрицу неточностей, граф концептуальной модели и так далее. Подробнее об этом можно прочитать здесь.

В этом разделе вы сконфигурируете свою среду так, чтобы TensorBoard отображался в Jupyter. Сначала вам нужно загрузить расширение блокнота tenorboard.notebook:

Теперь вы определите обратный вызов TensorBoard с помощью модуля tf.keras.callbacks.

Перестройте модель, используя Sequential API tf.keras:

Обучение и тестовые наборы были модифицированы для различных целей. Так что будет хорошо, если разделить их еще раз:

Вы все готовы обучать модель:

Панель инструментов TensorBoard должна быть загружена в Jupyter, и вы сможете отслеживать метрику обучения и валидац

How to train a tensorflow probability bijector (weights) in tensorflow 2.0?

The following example is a simple version of some bijector transform (for example a generalization of BatchNorm (fit a distribution to the inputs and use the CDF to transform the inputs like a copula would). The code below leads to «No gradients provided» errors.

Is there a pattern for this in the docs? I am not finding it.

1 Answer 1

Currently, tfb.Affine converts its constructor arguments to tensors (which loses the Variable gradient linkage, since not under a GradientTape ) immediately; we are working through the bijectors to defer this conversion, but Affine is a complex one.

In the meantime, AffineScalar is already done, so here is a version that works:

Добавить комментарий