To address the problem of mining potential melanoma metastasis biomarkers in pathological images of primary melanoma using deep learning methods, this work proposes a multimodal-based method for the diagnosis of melanoma metastasis conditions. The method consists of an attention mechanism-based biomarker mining network, a CNN-based infiltration depth prediction method, a unique thermal encoding-based patient information transformation, and a fully connected layer-based classifier. The method takes the pathological image of primary melanoma and patient information as input, extracts pathology image features through biomarker mining network, extracts infiltration depth features from pathology images by infiltration depth prediction module, and encodes infiltration depth and patient information via one-hot coding. Finally, the concatenated coded data and image information are diagnosed by fully connected layer taxonomy. To extract features from large scale pathology images more adequately, a biomarker mining network based on attention mechanism is proposed so that the model is more prone to focus on the more important image blocks among them when processing a large number of image blocks of WSI. In order to introduce infiltration depth as a predictor, a CNN-based infiltration depth prediction method is proposed. And the method of segmentation by classification is employed to measure the depth of infiltration employing and improve the prediction accuracy, which is a countermeasure to solve the problem of only coarse-labeled training data. The experimental results indicate that melanoma metastasis can be more effectively diagnosed by the proposed multimodal method than by the unimodal diagnosis method, with a high accuracy of 87%. As obtaining a higher prediction accuracy than the unimodal model and existing related studies, and being able to explore potential biomarkers related to metastasis in pathological images, the proposed method provides a significant guidance for the selection of clinical treatment plans for melanoma.