How to Deploy a Tensorflow Model to Production 

Siraj... my dude, yours are the shoulders I am standing on in my job. Thank you so much for all the incredible tutorials and additional resources!!!

I dived into the world of ML using scikit-learn and now I am learning the Tensorflow. I searched alot about the deployment of models, but I am having a hard times understanding the whole meachnaism. I really appreciate your effort, this is the best content on ML deployments on RU-vid 👍🏻

I made a model in Keras. Installed Keras and Tensorflow on AWS instance in Virtualenv and created single python instance listening to RabbitMQ with Pika and used Flask over WSGI to put messages to the queue. HTML client uploads a photo and is returned with ID than it should request id info from the server in one second intervals. Works fine and queuing is kind of bullet proof since it's running on a small cpu instance :)

Man, you are really underrated! You deserve a lot for these great videos about ML. A big thanks from Brazil for the awesome work!!!

I always wondered "Ok, I created a model, now what?". Thanks, Siraj!

Long story short siraj earned around 864.84 dollars for this month lol go to 35:40.. He deserves lot more.. Keep up the good work.

Hey Siraj, Firstly, great video Request: A tutorial on how to build a deployed Deep learning model that learns from live data and updates itself to a new version.

Thank you, I have been wondering what to do with a model once trained. No one seems to be talking about this and it seems like a very important step. And yes, I have been searching furiously to figure it out. Thanks again.

Loved the video, love the energy, he clearly has a deep understanding

Siraj, excellent tutorial - thanks for creating this.

Hey Siraj, Firstly, great video Request: Can u post a tutorial on tensorflow distributed training on GPUs and Kubernates.

Thanks for sharing this. I've been wondering how to do this.

Love your teaching :) Keep it up☺

Thanks so much for this vid! Could you by chance in the future do the same thing, but for something custom like a tensorflow model that simply adds two floats and returns the response? Reason I ask is because I have been having a big problem trying to figure out how to setup a custom model for serving with regards to configuring the proto files and client.

Thanks for this. It works for me.

I was very surprised by the fact that bazel have been building my custom client for serving for a very long time ...Does it need to compile so many sources every time when I change the client code?

24:18 When I run the command: 'docker build --pull -t $USER/tensorflow-serving-devel -f tensorflow_serving/tools/docker/Dockerfile.devel .' I get an error: 'invalid argument "/tensorflow-serving-devel" for t: invalid reference format' Help? (On Windows 7, Docker Toolbox) UPDATE: The following does work: 'docker build --pull -t tensorflow-serving-devel -f tensorflow_serving/tools/docker/Dockerfile.devel .'

If I have an existing tf model, and I trained my data with train_test_split, what to do if I want to see the predictions for my model on the entire dataset (including the original training and testing data)?

So cool! This is inspiring me to make some models. :)

Hi Siraj I have trained the keras model using imagdedatagenerator and flow_from_directory. When I deploy in tensorflow servimg i got wrong class due to shuffle true in flow_from_directory. How can i resolve this problem ? Thanks

His notebook is Untitled... He chose the dark path....

Your explanation was on point!! Thank you Siraj

"we have to deal with C++" count me in :DD

Great topic! Thanks Siraj.

lol need this in about two weeks thanks for a dank upload siraj!!!! really hope I dont run into that docker problem you had, I have over 20 docker images I think. lol 27:00 building custom linux kernels amirite lol

Very useful. Thanks Siraj!

such a helpful video!

This is amazing, thanks Siraj. Since we are talking about production level can you plz suggest server configuration required for Tensorflow serving?

@Siraj : Can we train the models and deploy them Incremental ?. Is TF Serving supports multiple smaller models. If yes, how can we do it. I cannot get any help in internet.

i am working o n image classificattion and my model is in tflite, how do i deploy? do i need to change anything in your video tutorial?

Thank you for these. I've learned a lot already. The big question and use case that I'm interested in is using ML in blender. The goal would be to create a blender add on that could be trained on and manipulate mesh in a character model. With Blender and its add ons all written in python, this seems doable. The mesh data can be called within the blender python api pretty easily. My question is how to best set up a system that would take a character mesh (this would be in the thousands of vertex coordinates) and then train a model on with shape keys for happy in each one, then be able to make a shape key on a new character mesh that also produces a happy expression.

Great video, Siraj! Could you add resources on how to deploy Keras models?

Hi Siraj, thank you so much for the videos. bazel build failed on some error and I am gonna try it tomorrow (it might be problem with not enough memory for docker). However I am kinda lost with docker and containers. Now when I shut it down, how do I get back to the step where I can write bazel bild etc..? Thank you.

Hi, How could we use this approach of deploying a TensorFlow model to production, for a re-trained inception model using transfer learning?

If I use a saved scaler function from sklearn for the input data - can that be loaded to the server along with the model? Basically before model is called - i have to use that function first on every input. I had to use a scaler since i have many inputs and they are very different : one can be in a range of 1-3, another 50000-1000000. For that i used StandardScaler from sklearn and it does great. In case of getting right prediction i have to apply it on the new coming data.

Hey siraj, actually i want to know that i have created a image classifier in docker and now i want to integrate into system which is written in python, how i can do that?

Hi @siraj, Could you please tell How to Deploy a Keras Model to Production?

May the force be with you

I barely ever comment on videos but I have got to show love for all I've learned on your channel. I've been recommending you to everyone I can find. What software do you use to do the screen background with you in the foreground by the way? I want to start a channel teaching atmospheric science and I like this style...

thank you for tensorflow video

How would one go from building tensorflow in docker on a local CPU without CUDA support and then deploying the container to a GPU instance in the cloud with CUDA support? Would I need to build tensorflow again when I deploy the docker container to the GPU and just enable CUDA support there? Or is there a way to have CUDA support on my CPU and maintain that when I deploy the container?

super informative:)

How can we take that JSON output and have it display bounding boxes in the browser?

When I run "bazel build -c .." , I get "no targets found beneath' tensorflow_serving' ".

How can one do incremental training of models already deployed to serving?

Hey Siraj! Thanks a lot for all this amazing content. I am working on generative models for symbolic (MIDI) music sequences. Your videos on the topic have been very useful. However, I'm intending on running the inference / generation part on mobile device (iOS). I am using TensorFlow and things seem to gradually improve (more functions, more support, more documentation) but I still find it very tricky to port the model on device (strip the unused / unsupported nodes, optimize, porting the generation scripts etc.). Even porting the fairly simple RBM model you used for one of your videos is challenging. Any suggestion on that? Given that running inference on mobile devices is becoming a trend, would you care to make a video about it?

Stupid question: Are we using the MNIST model? Since we're using the ImageNet model right?

I honestly dont get how to employ the model. At what stage do we use the neural net to make decisions about actionable data, in this case see it decode the words?

I don't get this part: 32:08 How that bazel command worked out of the docker?

Does this mean that in order to make an ML web app I don't have to learn Django or any other python web framework, I could just use tensorflow?

Nice vid man ! Did you like C (didnt see any vids about it :D). I would like to know more about Tensorflow C API. Thanks alot for all knowledge you share

Hey Siraj, Love the videos. Question: I am taking the Udacity DL course, and am getting more and more into it and plan to continue on to make a career out of this. Would you recommend I switch over to Ubuntu as my primary OS or is it feasible to stay in Windows?

very good video

I guess you have shared enough knowledge for someone to start a company :/

how can you upload new models at runtime?

the process crashes when i try to create the docker image, it goes on for about 10 minutes and then everything freezes. any idea why ? i have an intel i5 processor

Siraj, Please, put together a video tutorial on how to generate an Image from Text with TensorFlow. (Text to Image)

Siraj, could you please upload weights of models you trained as well, as neither online and personalized training of models is a viable option financially.

Siraj, if I wanna write an ML algorithm and make a web app based on it, would learning Django be useful for this task?

Thanks for end 2end tensor flow tutorial. World wows you for AI revolution

Siraj, I wonder if a similar strategy can be used to deploy a sci-kit learn model?

Hey Siraj, what software do you use to make those images like on 2:34 or 11:46 etc?

What’s the advantage of serving model using TF serving compared to a rest api

what's the difference between a docker image and a simple executable file?

Was waiting for this

Hi Siraj, nice video! But I'm a bit confused about classification_signature and predict_signature in MNIST example. Should I use both of them, is there any difference between them, why classification's input is a string, etc. Or it's just example that I can use number of signatures to query single model? Thank you.

Hello. I hope you will answer as it's really important for me. I'm currently working on a project and my task is to generate meaningful unique text from a set of keywords. It doesn't need to be large, at least a couple of sentences. I'm pretty sure I have to use LSTM but I can not find any good examples of generation of meaningful texts. I saw a few of randomly generated but that's all. I would be grateful for any advice. Thank you in advance.

I am currently facing a problem to deploy my custom models where my images data is located on my local computer, can u make a video to how to deploy custom models where image data is located in the local computer, save models and deploy for in android devices.

Can you please explain how I can make and serve CNN model using deeplearning 4j in java ?

i am not able to find step by step procedure to how to work on tensorflow in android studio

Please share link of part 2 of this tutorial for pushing this to cloud.

Hi Siraj, you mentioned at 28:00 that inception has 100s of layers. If I am not wrong presently it has only 48 layers. Please correct me if I am wrong or you are referring something else.

I am having issue while deploying my deep learning model in h5 format on flask. It works fine on local machine however, it has issues on my pythoneverywhere hosting server.

To build the docker file use : sudo docker build --pull -t $USER/tensorflow-serving-devel -f tensorflow_serving/tools/docker/Dockerfile.devel . To run : sudo docker run --name=tensorflow_container -it $USER/tensorflow-serving-devel

Come back buddy, we miss you!

There is no /tensorflow folder to do 'configure' as Google has taken it out. It is no longer required to do the configure according to Google latest issue response. Just do 'bazel build -c opt tensorflow_serving/...' at tensorflow-serving directory. (of course without the ')

Why we want to train the model.. I want deploy in website my chatbot.. Can you tell me

Does anyone have the github.com/tensorflow/serving/ repo when this video published as the latest version is deficit some files and its hard to follow form there. Thanks.

Hello Siraj, Firstly thank you for this great video. I train a model in Python, than i saved with the train.saver to generate my checkpoint. i want to load this model in C++ , is there a way to do that ????

Awesomeeeeee. Hello guys, do you know if is possible using: • Variational Autoencoders Neural Network (VAE) or • Generative adversarial networks (GANs) For structured data? I have seen some examples and it could be used but just for unstructured data such as images, audio, etc. Maybe do you have any example with structured data? Thanks a lot

Any video on image localisation

Please rehearse more and these would be 10000% better

bro, where is your untitled.ipynb i can't find it. please help.

Has anyone got an error like this, at the bazel build step? (run on Macbook Pro, OSX 10.11.6, via Docker method. With bazel 0.5.4 in Dockerfile) ERROR: /root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/external/org_tensorflow/tensorflow/core/kernels/BUILD:2904:1: C++ compilation of rule '@org_tensorflow//tensorflow/core/kernels:conv_ops' failed (Exit 4). gcc: internal compiler error: Killed (program cc1plus)

So sad this tutorial is broken now, they changed the github repository. Now you don’t have the tensorflow folder inside serving. If anybody knows of a tutorial please let me know.

Where would i be without you?

I think an alternate would be deploy the models in your code directly rather than calling a rest API. I have a model running on my iPhone, I don't see performance issues. The new chipsets are getting more and more powerful.

Siraj can you please please recommend me a book to get start on ML, your videos are great but I am having hard time in grasping the concept

#update The tensorflow submodule has been removed. You should no longer have to run TensorFlow's configure script manually

after uncompress the inception model, do --> 'bazel-bin/tensorflow_serving/example/inception_saved_model --checkpoint_dir=inception-v3 --output_dir=inception-export' as the command on the tutorial is old and no longer works.

the only useful video that you have uploaded till date!

ok ive been convinced.... i will stop being a stubborn js scrub... *sigh* welp time to learn tf

pls can i have a copy of your pipeline or pls how do i do mine?

bazel-bin/tensorflow_serving/example/inception_export: No such file or directory Now what?

"If Google can use it, we can use it." is so 2017.

Hi Siraj, Thanks for great video. please help me out to fix the issue,I have my own model. here i am using faster_rcnn_inception_v2_pets.config architecture. currently i have trained check points. But when ever i am exporting checkpoints by using below command bazel-bin/tensorflow_serving/example/inception_saved_model --checkpoint_dir=my-model6 --export_dir=inception-export at that time i am getting below error DataLossError (see above for traceback): Unable to open table file my-model6/model.ckpt-21292: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator? [[Node: save/RestoreV2_34 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_34/tensor_names, save/RestoreV2_34/shape_and_slices)]] Here we have TF=1.4 and Bazel=0.5.4 while training i got checkpoints like model.ckpt-21292.data-00000-of-00001 model.ckpt-21292.meta model.ckpt-21292.index for the above checkpoints i was renamed like model.ckpt-21292. I was followed your video, your downloading pre-trained model. but my question is we both having the same type of checkpoints, then why am getting above error?? Thank you

The day he can show proficiency in linear Algebra and differential equations etc, I would consider him to be a data scientist .. otherwise it’s all smartness practiced with code available in public

Someone should invent Keras for Tensorflow Serving

We wait for "How to tune hyperparameters"?

failed: gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 106 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 4. :( I can't get rid of this error, I'm new in ML/DL and that's what I get as my welcoming message, can anyone help please ?