OPEN JOB POSITION - POST-DOCTORAL SCHOLARSHIP ON COMPUTER VISION AND NATURAL LANGUAGE PROCESSING - USP - SÃO PAULO - BRAZIL

 

Knowledge area: Computer Science / Computer vision

Project Title: New methods for image captioning: a framework based on computer vision and natural language processing 

Specific area: Computer vision

Job start: November 2022

Supervisor: Roberto M. Cesar Jr.

Institution: University of São Paulo  - USP

Address: Rua do Matão, 1010, Cidade Universitária, São Paulo, Brasil.

Deadline for application: August 26st 2022

 

Inscription e-mail: thematic.data.science.usp@gmail.com

Subject: Application to POST-DOCTORAL SCHOLARSHIP ON COMPUTER VISION AND NLP

 

Description of the project

 

Scene description is a process that aims to associate one or more textual sentences to an image. Concerning urban infrastructure, this is a technique that allows the description of urban scenarios, such as the characterization of sidewalks in terms of size, the identification of objects and the positional relationship they have with each other, among others. In recent years important advances have been achieved in this task through the use of deep neural network techniques. These advances are facilitated by the greater availability of GPUs and large data sets.

 

Furthermore, advances achieved by neural networks have been mainly obtained in individual modalities such as vision, language or sound. In many cases, real-world problems have components that are embedded in more than one modality - as can be the scenarios of Urban Informatics.

 

In this sense, this project aims at developing computational methods to improve the processes of description of urban scenes. As object of study, we intend to explore remote-sensing and street-level images. The project will involve the use of computer vision techniques associated with natural language processing. The main idea is to use the textual language framework to improve the generated descriptions.

 

About the institution

The University of São Paulo is one of the best ranked Universities in Latin America, being considered one of country's more prestigious educational institutions. The Data Science Group at IME-USP is a traditional machine learning research working on the field for more than 20 years with strong international collaboration.

 

The selected candidate will be funded by a FAPESP fellowship with the one of the following conditions:  Initial funding of 1 year, fellowship R$ 90K per year (approx. US$ 24K / year PPP) plus overhead for travel expenses such as attending to conferences.  More information is available at http://www.fapesp.br/en/5427 The grant may also cover expenses for moving to São Paulo/Brasil (including flight tickets).

 

Required Skills

PhD degree with strong background in mathematical modelling and programming (e.g. Computer Science, Engineering, Physics, Math). Research experience and publications in one or more of the following areas: image processing, computer vision and natural language processing. Oral and written communications skills (English).

 

Application details

Please send the following documents to:

 

Inscription e-mail: thematic.data.science.usp@gmail.com

Subject: Application to POST-DOCTORAL SCHOLARSHIP ON COMPUTER VISION AND NLP

- CV

- Link to ORCID, Google Scholar, or ResearcherID

- Summary of doctoral thesis and other relevant works

- Two recommendation letters from former supervisors or collaborators.

 

 

References

 

[1] Simao Herdade et al., Image captioning: Transforming objects into words. Em: Advances in Neural Information Processing Systems, 32 (2019).

 

[2] Aishwarya Kamath et al., MDETR - Modulated Detection for End-to-End Multi-Modal Understanding. Em: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Out. de 2021, pp. 1780– 1790.

 

[3] Duo Wang e Salah Karout, Fine-grained Multi-Modal Self-Supervised Learning. Em: arXiv preprint arXiv:2112.12182 (2021).