SyllaBUS – 6670

Learning Outcomes

After taking this course, you will be able to:

·      Define precisely the problem statement for “core” computer vision problems, in terms of the expected input, expected output, and metrics for measuring performance. These core problems include:

o   Recognition - Image Classification

o   Recognition - Object Detection

o   Recognition - Semantic Segmentation

o   Recognition – Pose estimation and other structured prediction problems

o   3D Reconstruction – Stereo

o   3D Reconstruction – Structure from motion

o   Image synthesis – generating realistic images

o   Image synthesis – style transfer

o   Embodied vision – End-to-end learning

·      Describe and implement new vision architectures such as transformers and convolutional networks

·      Explain the basics of end-to-end learning and reinforcement learning.

·      Derive the mathematical equations underlying perspective image formation

·      Describe the physics underlying image formation intuitively, and with certain assumptions, precisely.

·      Explain the key technical challenges that must be solved for each of the above problems

·      Describe how these technical challenges are currently solved

·      Describe intuitively the current state-of-the-art: what current techniques can and cannot do

·      Describe intuitively capabilities that are missing, extrapolating beyond the above established core problems

·      Execute part of a research project:

o   Define concretely a research problem, describing the problem setup (i.e., input/output and metrics), as well as why the previous work is lacking

o   Identify a concrete direction of enquiry for tackling this problem.

o   Propose and perform a “proof-of-concept” for this direction

o   Articulate what additional experiments need to be done to confirm conclusively the merits of the proposed approach (or lack thereof)

o   Put together all of these into a concrete 2 pager of proposal + initial results.

·      Perform peer review: critique a peer’s proposal, providing constructive feedback about past work that might be missed, important questions that might be unanswered, and further suggestions to improve this direction.

 

Choosing a Research Project

If you are a MS/PhD student working in computer vision, your class project SHOULD BE related to your research

If you are a MS/PhD student who does not work in computer vision, try to build a project that is relevant to your research interests

 

Scope: The project in this course need not be a full project; we will only aim for a “proof-of-concept” in terms of results. So choose a project where (a) the proof of concept is achievable within the time frame of a semester, and (b) the proof of concept should be useful for a broader, more important research question.

An important consideration here is also compute. Choose a project where the proof-of-concept can be shown using a single Google Collab notebook.

Given these considerations, do not choose a project where the main idea you are exploring is “scaling up”.

Novelty: A key aspect here is that we want this to be a research project. This means that there must be some underlying question that has not been answered. Thus, simple re-implementations are not good projects.

Note that the research question itself may not be a computer vision question. For example using an existing CV approach on a new domain is a valid question.

 

Deliverables

 

You will have the following deliverables:

1.     December 5: A one pager that answers the following two questions for each of the four sections of the course: Recognition, Reconstruction, Synthesis and Embodied vision

a.     What are the open research problems, namely, things that current state-of-the-art cannot do?

b.     What are the technical challenges in solving these problems? Brainstorm about possible solutions.
For both of these, be creative! More points for thinking out of the box.

2.     A project with three deliverables:

a.     October 7: A one sentence project idea

b.     November 11: A two-page project proposal that contains:

                                               i.     Introduction that motivates the particular problem you are working on

                                             ii.     Related work that clearly describes what has been done and what is missing in prior work

                                           iii.     A section describing your approach and how it addresses the limitations of past work

c.     December 5: A one-pager final result

                                               i.     A preliminary result showing evidence your approach might be successful.

3.     November 22: Peer review for 2 project proposals from your peers (Papers will be assigned to reviewers by November 15). This peer review should answer:

a.     Is the problem well defined?

b.     Does the related work clearly identify the holes in the prior work?

c.     Does the proposed approach address the limitations of past work?

d.     Do you agree with the authors conclusions from the preliminary experiments?

e.     What further experiments and modifications to the approach would you suggest the authors do?

 

GRADING POLICY

You will be graded based on your project, your reviews and your summaries.

The breakdown will be:

Project: 50

-       Clarity of problem: 10

-       Related problem coverage: 10

-       Approach: 10

-       Preliminary results: 20

 

Reviews: 30

-       Q1 – 5

-       Q2 – 5

-       Q3 – 5

-       Q4 – 5

-       Q5 – 5

-       Constructive comments - 5

Summaries: 20

-       Creativity – 10 points

-       Correctness – 10 points

 

Course Management and Policies

 

Academic Integrity:  

Each student in this course is expected to abide by the Cornell University Code of Academic Integrity. Any work submitted by a student in this course for academic credit will be the student's own work.

 

Accommodations for Students with Disabilities:  

Students with Disabilities: Your access in this course is important. Please give me your Student Disability Services (SDS) accommodation letter and email me a note early in the semester so that we have adequate time to arrange your approved academic accommodations. If you need an immediate accommodation for equal access, please speak with me after class or send an email message to me and/or SDS at sds_cu@cornell.edu. If the need arises for additional accommodations during the semester, please contact SDS. Student Disability Services is located at Cornell Health Level 5, 110 Ho Plaza, 607-254-4545, sds.cornell.edu.

 

 

 

Inclusivity:

 Computer vision is a technology fraught with many ethical issues in its current practice. As new entrants into this field, you have the power to change this for the better. We can start by keeping our course an inclusive environment that supports everyone’s learning, maintains a civil discourse, and respects what every one of us brings to the table.

Mental Health and Stress Management Resources

 If you are feeling overwhelmed, or worried about a friend, please reach out to one of your instructors or your academic advisor.

Please look at this guide that collects all the resources that you can avail of.

 

Note that Cornell has trained counselors available to listen and help: Empathy, Assistance, and Referral Service (213 Willard Straight Hall, 607-255-3277), Cornell Health's Counseling and Psychological Services (CAPS, 607-255-5155), and Let’s Talk.