i am trying this paper in the caffe i am using NYU2 raw depth dataset. about 12K~13K sampled depth image. at the train task of you, did you use filtered image(cross bf or colorization) or not(raw depth image)? colorization is good filter method but it is too slow

Hello!

I am trying to recreate your results on the NYU_depth dataset with Pytorch. I am fairly confident that my network structure, loss function, and data augmentation process is correct, but I am unable to reach a similar depth image quality as your Tensorflow outputs (see the attached images).

My guess is that the difference might be in the training process. I tried to work according to your article, but a few details are unclear. You wrote that you gradually reduce the learning rate when you observe plateaus. How do you define a plateau, and what does gradually means in this case?

To get the results below, I used SGD optimizer with 0.01 init LR and 0.9 momentum, and I halve the learning rate after every 7th epoch.

some test image:

the output using your Tensorflow network:

the output using my Pytorch network:

Hi,

I am trying to replicate the output of your network, I have created my training code based on the details your provide in the paper (data augmentation, learning rate init. 0.01, pretrained resnet50 weights, ignoring the invalid pixels, implemented the inv. huber loss, batch size 16 and 20 epoch). The results I got from the network look like this:

As you can see, the prediction of the network and the real GT look very similar from the geometrical perspective, but it seems they are in a different scale or the network has learned from images with very low resolution and has done an upsampling (because the values are the black ones 990-999 mm and the clearest ones have values of 199X).

If anyone can bring some light to this please. Best regards.

I restore the model that you provide, and finetune the layers except the resnet50. I use Berhu loss. And I use AdamOptimizer to minimize my loss, the learning rate is 0.0001, and after 10000 steps the lr is 0.000001. The whole steps is 20000. Here is my loss curve: Here is the prediction that I use my model: As you see, the loss value is small, but the prediction is not very well, it is ambiguous. Can you give me some advice? Thanks! @iro-cp

Hi, @iro-cp , I want to fine-tune from your nyu checkpoint model for my own dataset.

This is my fine-tune loss (berhu as your paper), fine-tune code is similar with tensorflow-deeplab-resnet Finetune layer : layer16x_ and ConvPred Learning Rate : 0.001 BatchSize : 8 Optimizer : AdamOptimizer

Tensorboard shows the depth result doesn't smooth in plane scenes (such as wall, red circle)

Can you provide some advise? Thanks very much for sharing this perfect work!

Hello, I have a little problem, I try to test the image on preduct,py. When I input commend "python predict.py NYU_ResNet-UpProj.npy (my own image)", it's get a error like this:

How to fix it problem? Thanks you!!

In network.py:

  def prepare_indices(self, before, row, col, after, dims ):
    x_0 = tf.Variable(x0.reshape([-1]), name = 'x_0')
    x_1 = tf.Variable(x1.reshape([-1]), name = 'x_1')
    x_2 = tf.Variable(x2.reshape([-1]), name = 'x_2')
    x_3 = tf.Variable(x3.reshape([-1]), name = 'x_3')

This is the wrong use of tf.Variable (variable nodes are used as a source for value that the net expects to change, typically weights). In this case, this is just reshaping the indices from np.meshgrid, so these values aren't weights, or anything like that. There could be a specific reason these are made as tf.Variable that I'm unaware of, but it seems these lines should be:

  x_0 = x0.reshape([-1])

Why this matters: tf.Variable nodes typically store "trainable" values, which must be stored in checkpoints and loaded weight files. Since these are four 4D-flattened-to-1D arrays and there is a set of these for each up-conversion, this is a lot of data being stored to disk, which must also be loaded from disk (and saved, when creating checkpoints). These are basically indices so no change (learning) is expected, this saving and loading I propose is needless.

Case in point, these variables seems to be the prime contributor to the long load time of the weights file. In predict.py:

  net.load(model_data_path, sess)

takes several minutes on my computer in the current state. Changing prepare_indices() as indicated above reduces the load time by orders of magnitude, however making this change MIGHT make the new model incompatible with the current weights file, NYU_ResNet-UpProj.npy (I am having trouble making the net work with this change, so more investigation is needed on my end, but I figured I would raise this issue in case others are available to work on resolve this).

Since this is a non-functional change, I propose the authors try the following:

1. Remove the tf.Variable nodes as shown above (making them simple operations)
2. Retrain using identical meta-parameters as in the paper (if the starting weight values are still available)
3. Compare results pre- and post- change to ensure they generate the same output?

If the starting weights aren't available, I suppose a full retraining would just need to generate acceptable results.

Hi, I found websave function is not exist in my matlab R2014a. So it will result in error when websave is used in .m file. What can I do to solve this problem？

I am trying to train this model using your TensorFlow code. But it can't converge. I am using 'nyu_depth_v2_labeled.mat'. I accept the L2 loss for convenience. The raw depths are used. Invalid pixels (where depth is zero) have been excluded from training. I have tried to fix the ResNet50 or not.

The code is as below:

# NetWork
graph = tf.Graph()
with graph.as_default():
    # Create a placeholder for the input image
    tf_train_dataset = tf.placeholder(tf.float32, shape=(None, img_height, img_width, num_channels))
    tf_train_labels = tf.placeholder(tf.float32, shape=(None, depth_height, depth_width))

    # Model.
    # net = models.ResNet50UpProj({'data': tf_train_dataset}, batch_size, trainable=True)
    net_ResNet50 = models.ResNet50({'data': tf_train_dataset}, batch_size, trainable=False)
    layer1_BN = net_ResNet50.get_output()
    net = models.UpProj({'layer1_BN': layer1_BN}, batch_size, trainable=True)

    # Training computation.
    output = tf.squeeze(net.get_output(), squeeze_dims=[3])
    loss = tf.reduce_mean(tf.nn.l2_loss((output-tf_train_labels)*(tf_train_labels!=0)))

    # Optimizer.
    global_step = tf.Variable(0, trainable=False)
    starter_learning_rate = 10**-3
    learning_rate = tf.train.exponential_decay(starter_learning_rate, global_step,
                                               200, 0.8, staircase=True)
    # optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)
    momentum = 0.9
    optimizer = tf.train.MomentumOptimizer(learning_rate, momentum).minimize(loss, global_step=global_step)

    # Add a scalar summary for the snapshot loss.
    tf.summary.scalar('loss', loss)

    # Build the summary Tensor based on the TF collection of Summaries.
    summary = tf.summary.merge_all()

# Train
with tf.Session(graph=graph) as session:
  tf.global_variables_initializer().run()
  saver = tf.train.Saver()

  # Instantiate a SummaryWriter to output summaries and the Graph.
  summary_writer = tf.summary.FileWriter(log_dir, session.graph)

  # Load the converted parameters
  print('Loading the model')
  net.load('NYU_ResNet-UpProj.txt',session)
  print("Initialized")

  for step in range(num_steps):
    # Pick an offset within the training data, which has been randomized.
    # Note: we could use better randomization across epochs.
    offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
    # Generate a minibatch.
    batch_data = train_dataset[offset:(offset + batch_size), :]
    batch_labels = train_labels[offset:(offset + batch_size), :]

    # Prepare a dictionary telling the session where to feed the minibatch.
    # The key of the dictionary is the placeholder node of the graph to be fed,
    # and the value is the numpy array to feed to it.
    feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
    _, l, predictions = session.run([optimizer, loss, output], feed_dict=feed_dict)

    if (step % 10 == 0):
      print("Minibatch loss at step %d: %f" % (step, l))
      print("Minibatch accuracy: %.1f" % accuracy(predictions, batch_labels))
      # Update the events file.
      summary_str = session.run(summary, feed_dict=feed_dict)
      summary_writer.add_summary(summary_str, step)
      summary_writer.flush()

  # print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))

  saver.save(session, model_path)

  print('Done!!!')

The loss curve is as follws:

line 46, in setup .conv(1, 1, 512, 2, 2, biased=False, relu=False, name='res3a_branch1') ValueError: ('stride must be less than or equal to filter size', 'stride: [2x2] filter: [1x1]')

time comparison on a single image (inference): Elapsed time for run: 0:00:02.076628 (pack + reshape) Elapsed time for run: 0:00:02.555231 (dynamic stitch)

disclaimers:

• I ran on a CPU instead of a GPU, so it's probably worth testing that.
• I included the timing code so you guys know what I did to time it (just wrapped the session.run)
• I included a get_incoming_shape since I just felt like lists with integers are easier to deal with than tensorflow's Dimension stuff.
• Also added code to display the input image. Thought it would be nice to compare.

I get an error everytime I try to run the prediction on several images through a jupyter notebook.

Do you know how I can fix this?

I think it's better if you define a function load_model in predict.py to create model and load weights from the checkpoint and returns the model that could be passed later to the run function.

I'm trying to implement that but the tf.Session() seems to make it a bit complicated.

Hello and thak you for your work!

Testing your code for the NYU_v2 trained tensorflow models worked great for me! But I'd also like to test your model on outdoor scenes. So I installed Matlab and got it linked to MatConvNet but I cant't get it to run...

I am tryind to test the code with an arbitray image, therefore my command Window statement is as follows:

DepthMapPrediction my_picture.jpg Make3D_ResNet-UpProj.mat

Running this the error statement is as follows:

**Error using imresize>parsePreMethodArgs (line 379) Invalid input syntax; input image missing from argument list.

Error in imresize>parseInputs (line 273) parsePreMethodArgs(varargin, method_arg_idx, first_param_string_idx);

Error in imresize (line 152) params = parseInputs(args{:});

Error in DepthMapPrediction (line 40) images = imresize(images, net.meta.normalization.imageSize(1:2));**

Does somebody have any advice on where to go from here?

(Using Matlab R2020b, and matconvnet-1.0-beta25)

I am confused about I did get the same depth estimation pictures as yours,but I dont know how can i use it.it can get the truth distance between object and camera?if it can get the distance and can you show me the detail?

I see default input size is 304228,but use this to tensorflow savedmodel or mobile model,I need to specify the input size,always pic is not 304228,I scale it and fill it with rgb(255,255,255),but this will make result accuracy low,so how to adjust input image?

Actually I have prepared my own data set of indoor scene in my environment and want to train model on that. I am freezing all other layers except for the up projection blocks and the result is not so good. Even I trained it on as small data set as 600 images and achieved 82 percent accuracy but the results were not good visually. I donot know the reason of that maybe you can suggest me something. And the images I want to train are approximately 6k. The pretrained weights with NYU are even performing better. batch_size = 32 learning_rate = 1.0e-3 monentum = 0.9 weight_decay = 0.0005 num_epochs = 70 optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=learning_rate, momentum=monentum, weight_decay=weight_decay) and lr is halved after 10 epochs.

Validation depth image

rgb image